We refreshed the dremio cluster and logged in and found all of the reflections had been removed by dremio. Here is a profile for an example query that $dremio triggered to remove a reflection. Do you have any idea why this happened? This is in spite of Never Expire being set on the Source.
reflection_remove.zip (3.5 KB)
Also I noticed on a seperate occasion that the source Reflection settings had been updated without anyone doing so. Is it possible some of these settings are cached locally so rolling the cluster would clear them out?
Hi @kprifogle
Can you post the screenshot of the reflection refresh policy screenshot on the source as well as the dataset.
See if you have any API calls managing the reflections outside of Dremio UI
@Venugopal_Menda
Hi,
Reflection Refresh policy is set to “Never Refresh” and “Never Expire”, and incremental refresh is turned on.
I dont have any external api calls managing the reflections outside dremio. Also the $dremio user deleted the reflection so it seems like it was part of a standard daily clean up.
Kyle
@kprifogle
never refresh and never expire work only on reflections that have their setting at default, like attached image. It seems like changing to “Incremental Refresh” does override “Never expire”. Do you also see
“REFRESH REFLECTION” jobs on this dataset?
