Reflection dependency

Dremio University
1

Hi dremio admin,

my current dremio version is v.19x:

  1. can I modify the dependency for each reflection? since sometimes i would like to avoid the repeat run on the child reflection acceleration if it got multi parent dependency , i would prefer to make a “AND” condition to wait all the parent jobs finished together first before to kick start the child jobs to run

  2. can i custom choose which reflection materialization to pickup instead of using dremio planner if the running query does not hit the reflection as I expect?

thanks a lot!

2

any update? which version will support this? thanks

3

You can’t modify the reflection dependencies. The planner figures out for a given reflection what other reflections it can use to accelerate a reflection’s materialization. Once this dependency is established, Dremio ensures that all reflections are eventually consistent with the underlying tables. That said, we are making a lot of improvements in this area. One of which is to avoid duplicate refreshes by having child reflections wait for all parents to finish before refreshing itself. Refreshes can also get a lot more smarter if all source tables are based on Iceberg and we have snapshotIds to track data changes.

Regarding your second question, you can choose which materializations are used for a given query through the new reflection hints feature. See Influencing the Choice of Reflections Used to Satisfy Queries | Dremio Documentation

4

thx for reply, i understand the backend logic more
btw are there iceberg table format support for dremio ver.19.x
is it just like incremental load by detecting the field change using business key / primary key?
the changes detection not just for new records , but also for existing changed records? as i know there is a bug for capture the existing changed records in ver 19.x