Our system has its own metadata, how can we set our metadata to dremio’s metadata?
Can you provide related documents or interfaces?
THX
Can you say a little more about the kind of metadata you have and how it should be used by Dremio?
We want to be able to use the latest metadata in real time, instead of relying on dremio’s timer to refresh the metadata, is there any way to do this?
@JoyJava you may want to check out Metadata Caching settings on the source settings screen. These options are set individually for each source. The following should get you closer to real-time:
- Dataset Discovery - Fetch interval to 1 minute.
- Dataset Details - Fetch mode to “As needed”.
- Dataset Details - Expire after to 1 minute.
Note that this is not usually recommended due to query time performance considerations.
@can which version supports this metadata caching?or is this real-time solution only supported in Enterprise Edition.
my setting page displayed as following:
Dateset Discovery option is not displayed.
We’ve got it, and we found this statement in the source code,
SQL is as follows: ALTER TABLE “es-test”. “Xxxx”. “Default” REFRESH METADATA
Hope to be useful to you
1 Like
Is there also any way to set the refresh delay or expiration of a reflection / acceleration ?
eg i have : a source data table, let’s say country where the refresh is set to every day
another source data table, events, where the refresh is set to be every 2 hours.
I have an acceleration dataset that is calculation from country and events, and I want it to be refreshed only once per day, not more. Currently I don’t think there is any setting to define the reflection/acceleration refresh delay, isn’t it ?
@Isabella_Zong all editions of Dremio support metadata caching. As @JoyJava mentioned, you can use the ALTER TABLE X REFRESH METADATA command to force a refresh. Also, Dataset Discovery option does not apply for file-system sources such as HDFS, MapR-FS or NAS – as we retrieve file/folder listings for these source types in the UI live.
@dfleckinger you can configure Reflection refresh and expiration by going to settings for a physical dataset or data source. Note that settings on the physical dataset will overwrite the source-wide settings.
For joins, Dremio uses the smaller refresh interval as the refresh interval for that reflection.
It is important to note that reflection refresh is independent of dataset metadata refresh. Reflection refresh jobs will use whatever metadata is available for that dataset at the time of reflection refresh.