dremio ver: 19.0
As I have to append and update the daily delta change records to previous snapshot of table instead of full historical load. However dremio not able to perform insert select operation.
For workaround there is a “incremental load” option for PDS (just insert but not support update)
If the current design have 3 layers and I choose the incremental load for fast performance and build reflection for both vds #1 & vds #2
PDS > VDS#1 (remove the duplicates of old records for same id) > VDS#2 (agg.)
will both reflections run on a incremental basis or full basis?
will vds#2 inherit the incremental load? will the incremental just apply to vds#1?
Would there some workaround to apply incremental load since takes tons of time to do the full load. Many thanks for solving my pain point!
hi any idea ~ thanks a lot
@crazyisjen Have you thought about Iceberg table format so you can do DML?
Yes I can try the Iceberg table, can ver.19 can support DML? as I tried seems cannot run insert/update statement to the table
When detecting the source file data is changed, will there a way to have a SCD2 capture (store historical data ) automatically after setting the business key and changing column in the configure page?
If the S3 file was not convert into iceberg table, can I still perform DML?
Thanks a lot!
@crazyisjen V19 does not support DML, best is to be on the latest 24.x
Any reason you are on 19.x?
okay let me try to upgrade to 24.x
is that need to convert the s3 files into iceberg table first before perform update/ insert? If for PDS (import from database source table), is it also can perform DML?
Any aspect of limitation of using iceberg table
thanks
@crazyisjen First upgrade and then convert