I am trying to implement incremental reflection in Dremio for a specific use case.
I created a view on the table using the following query:
SELECT ID, Cust_Sub_Id, Msisdn, Event_Date, Revenue
FROM mysql16.EDA.“dremio_reflection_test”.
A ROW REFLECTION was then created on this view.
Next, I executed the query:
SELECT Cust_Sub_Id, Msisdn, Revenue
FROM “Reflection R&D”.“VDS 001”.
Afterward, I inserted 2-3 new rows into the table and refreshed the reflection. Upon re-executing the same query:
SELECT Cust_Sub_Id, Msisdn, Revenue
FROM “Reflection R&D”.“VDS 001”.
I expected the system to scan only the newly added rows in the table. However, when I checked the Rows Scanned section in the job details, I observed that all rows were being scanned instead of just the new ones.
Does Dremio support scanning only newly added rows in this scenario? If so, how can I achieve incremental reflection based on this use case?
Also I have an another doubt that In Dremio, that does the reflection refresh process only add newly inserted rows to the snapshot, or does it take a snapshot of the entire table again after each refresh?