External_query() reflection refresh schedule - what is it?

[capnjosh]

I have Virtual Data Set that uses external_query() to make sure Snowflake does the heavy lifting rather than Dremio downloading an entire table and then doing some simple transforms and groupings.

When would a Reflection on this Virtual Data Set be refreshed? According to the “Reflection Refresh” configs on the data source used by external_query() or something else? Some observations follow:

I can manually refresh the reflection on the Virtual Data Set by disabling the reflection, hitting Save, then enabling the reflecting and hitting Save.

The data source used by external_query() has “Reflection Refresh” configs, but they always reset to “Refresh every 1 Hour”, even if I change it to a once-per-day schedule and hit Save. Am I missing something that might be overriding that config?

I wish I could just schedule a reflection refresh on the Virtual Data Set directly, regardless of any Physical Data Set refresh schedules.

I’ve seen the following discussions, and it seems like a “refresh Virtual Data Source reflections” feature would be a good one. I’m guessing it’s one of those things that isn’t that simple, but I’d like to vote for it.

Automating Reflection Refresh

Hi, We have a standard use case with Dremio on top of parquet files on S3 serving Tableau dashboards. Since there are many concurrent users and the dashboards are complex, we have to make heavy use of reflections. The problem is how to automate reflections refresh. There are 3 options: Refreshing a physical dataset (PDS) using the API (Dremio) refreshes all downstream reflections in the correct order. The problem is that if a reflection uses 10 PDS-s, it will be refreshed 10 times, which is…

Refreshing reflection of VDS never triggered

Hi - I have a VDS which is running a rather long query. I want to use reflection to store those results, but at the same time I want to be able to update the results. How can I do that? There seems to be no option to refresh a VDS through an sql command. At the same time I see no option to set a refresh schedule (like for a PDS reflection). I tried updating the metadata of all the datasources that are being used in the query (with ALTER PDS … REFRESH METADATA FORCE UPDATE’), but that doesn’t…

How is the VDS's reflection refresh scheduled?

Hi there, Let say I have a VDS, created from many different PDSs. Some of those PDSs have their own reflection refresh policy, or maybe all of them are set never refresh So if I create a reflection on the VDS, how is that reflection refresh scheduled? I visualize both cases for easier understanding as below [image] [image] Thanks,

[Benny_Chow]

@capnjosh

According to the “Reflection Refresh” configs on the data source used by external_query()

Yes, that’s how it works. If your Source Settings → Reflection Refresh → Refresh Settings is set to every hour, then the reflection should refresh every hour.

If you are not able to change that to 24 hour, that seems like a bug.

You could also script the reflection creation and deletion with SQL.

ALTER VIEW CREATE RAW REFLECTION XYZ

ALTER VIEW DROP REFLECTION XYZ

[capnjosh]

Thanks for the confirmation. Reflections that use external_query() are refreshed according to the reflection refresh schedule of the data source itself (rather than any specific table inside the data source).

What is the best way to report a bug? Just create a new post here? edit: I just created a new post about the data source reflection refresh settings not saving as expected.