Incremental reflection update question

1

If I use a modified date as the “Identify new records using the field”, and a record from the underlying data source is deleted, will the incremental reflection update also remove the deleted record or will it only add records with a modified_date > the previous max(modified_date)- leaving deleted records in the reflection?

2

Hi @summersmd

Only append-only datasets are supported for Incremental Update Mode. Updates and deletions of underlying files leads to incorrect results. Dremio recommends using Full Refresh in this case.

Incremental Reflections

Thanks
@balaji.ramaswamy

3

could you please give a link on how to use a modified date as the “Identify new records using the field”?i am not able to find “Identify new records using the field” in dataset settings -> reflection refresh. How do I make it appear? do I have to add a new column to the dataset/PDS as the modified date?

Thanks!

4

@gypsysunny,

Just answered your question here

Thanks
@balaji.ramaswamy