If I use a modified date as the “Identify new records using the field”, and a record from the underlying data source is deleted, will the incremental reflection update also remove the deleted record or will it only add records with a modified_date > the previous max(modified_date)- leaving deleted records in the reflection?
Only append-only datasets are supported for Incremental Update Mode. Updates and deletions of underlying files leads to incorrect results. Dremio recommends using Full Refresh in this case.
could you please give a link on how to use a modified date as the “Identify new records using the field”?i am not able to find “Identify new records using the field” in dataset settings -> reflection refresh. How do I make it appear? do I have to add a new column to the dataset/PDS as the modified date?