I have a VDS which is running a rather long query. I want to use reflection to store those results, but at the same time I want to be able to update the results.
How can I do that?
There seems to be no option to refresh a VDS through an sql command. At the same time I see no option to set a refresh schedule (like for a PDS reflection).
I tried updating the metadata of all the datasources that are being used in the query (with ALTER PDS … REFRESH METADATA FORCE UPDATE’), but that doesn’t trigger a reflection update either.
@cklar The reflection refresh should happen in the background automatically based on the PDS schedules, if your VDS is joined with 2 PDS and each one refreshes at a different frequency, Dremio will decide on the refresh frequency. It is also important that the metadata refresh is complete before the reflection refresh starts. An alternate way is to use the API Dremio
The id to pass here is any one of the PDS ID (from the above example), Dremio will then refresh the chain which also includes the VDS
@balaji.ramaswamy
Thanks for the response!
I was trying the refresh with the SQL command but a VDS update was never triggered.
One caveat is that I just refreshed the data sets without actually changing the underlying parquet files. Does Dremio recognize that the file is unchanged? That could be an explanation that the VDS didn’t change.
Is there any way to actually just trigger the VDS update directly?
@cklar If no files have been added, metadata refresh will skip unless you add FORCE UPDATE at the end, VDS is just a view and a query on top of another VDS and in turn on top of another PDS, so refreshing a reflection is a physical operation and is on the PDS dataset id