Reflection used with VDS query but not VDS reflection

1

Hi,

we have an incoherent behaviour in the reflection selection in the following situation:

Setup:

  • VDS1 is joining some large dataset (VDS0) and a smaller one using a join (VD2)
  • There is raw reflection on VD2 with all fields display et 2 sort, minimize file size + arrow caching
  • There is a raw reflection on VD1 on a subset of fields

Behaviours:

  • When running VDS1 (Preview or full) the VDS2 reflection is used
  • When running a refresh of the VDS1 reflection using on/off or catalog refresh of the VDS1 underlying PDS, the VDS2 reflection is not used (matched but not used with more information)
  • When running a refresh of the VDS2 reflection using a catalog refresh of the VDS2 underlying PDS, the VDS2 reflection is updated and the VDS1 reflection triggered BUT the VDS2 reflection is not used (matched but not used with more information)

So it seems that Dremio knows that the VDS2 reflection should be used during the VDS1 reflection refresh but it does not use it…
Is there a bug ?
Where can we find more details in the job logs why a reflection is not used ?

Tks & Rgs,
Alexandre

2

For the behaviors, here’s what I would suggest:

Second bullet, enable verbose profiles and review the best cost plans (in the acceleration tab) to see why VDS2 was not used. (Set support key planner.verbose_profile)

Third bullet, if updating VDS2 triggers a refresh of VDS1, then there should be a dependency between these two reflections. You can confirm this in the sys.reflection_dependencies table.