I see Dremio keep refreshing the reflection immediately after the prior one finishes:
This is a filtered job history view for the particular dataset(890a74b9-bb56-450f-b168-7ae93341bd60). The cycling lasts forever until I disable the reflection on this virtual dataset.
Do you have any idea what causing this? Thank you.
Hey @dli16, seems a bit off. What are the Refresh Policies you’ve set on physical datasets that the Balances VDS selects from?
For information, also experienced the same. No particular refresh plociy having a low Refresh delay.
It’s 8 hours to refresh and 16 to expire.
Hi @can
I’m also experiencing the same issue with one of my virtual datasets on Elasticsearch source. I have the Refresh Policy as “never refresh” and “never expire”. I’m using Dremio 3.0.6. I have many datasets with reflections however this one keeps refreshing indefinitely as soon as I create the reflection on the data set. I see “REFRESH REFLECTION” and “LOAD MATERIALIZATION METADATA” running one after another until I remove the reflection on the dataset.
Is there some way to check whats causing this?
Hi @igreg
We have to completely disable reflections to stop refreshing. Can you please send us “select * from sys.reflections” and send us the JSON download.
Also is there a possibility someone is refreshing reflections via a REST API?
Thanks
@balaji.ramaswamy
Please find the output attached. I needed to mask out the displayColumns field. No one is invoking refresh via the API. This only happens when I initiate a new reflection. The existing reflections are not being refreshed.
tmp-untitled.zip (2.7 KB)
Just wondering if you had a chance to look? Looks like the issue is sporadic. It happens for some elasticsearch indexes but not others.
Just wanted to share some findings. It looks like whenever a dataset contains a column with an empty list as the value, it causes the reflection creation to fail.
In dremio 3.0.6 this then causes the reflection to be retried indefinitely. In dremio 3.1.3 looks like infinite reflection retry does not happen any more however the reflection creation still fails.
Please see the screenshot attached for an example. The “bids” and the “offers” column is an empty array. If I remove these columns from the reflection it succeeds.
