set time for when to refresh, if the ETL load finishes at 11PM, we want to setup Dremio reflection to refresh after 11PM, is there a SQL command or some other way to specify the time
Currently, there is no way to specify a fixed time to start the refresh. The support of this feature is definitely in our roadmap.
One workaround is to enable the reflection at 11 PM on the first day so that it will run every day at starting around 11 PM or little later (depending on the workload of the system
can we refresh on demand, usually the ETL that loads DW (source for Dremio once a day), in case we run it more than once, how can we kick off the refresh of Dremio reflections. Any command, back-end, SQL or some other API to achieve this?
Currently, there is no way to trigger a refresh on demand.
This is feature is in our immediate road map.
For a “Full update”, you could achieve an “on-demand” refresh by disabling and enabling a reflection.
Since disabling a reflection deletes any previously cached data, this will not be useful for incremental update
Is there a system view which can be queried to see the last refresh date time for each reflection?
You can see the last refresh time of a reflection by clicking the jobs link of the specific reflection. The time of the last “CREATE TABLE” in the Job pages should give you the refresh date time.
Another way is to run a query “select * from sys.materializations where layout_id=<layout_id>”
The “create time” should give you the last refresh time.
You can obtain the layout_id for a reflection by clicking the jobs link of the specific reflection and copying the id shown in the job . See attached -
Please note that we have not finalized the sys tables and hence the table names and columns are subject to change.