I sometimes modify the database and hope to be able to query the newly added tables from the database in real time. Can I actively refresh the metadata instead of refreshing it every other time through the metadata refresh strategy?
Or can dremio trigger some events when I add tables to the database?
@balaji.ramaswamy
1 Like
You can try this command
ALTER PDS Alfresco.public.alf_node REFRESH METADATA FORCE UPDATE
@unni If your data is on HDFS or Hive, FORCE UPDATE is not required as the default without anything called LAZY UPDATE, Dremio will skip the refresh when there are no changes to the promoted folder while in FORCE UPDATE, it would not do the timestamp check. If data is on S3, the 2 commands are the same
@unni @balaji.ramaswamy Thanks, I will try this way.
@balaji.ramaswamy @unni
This method seems to only refresh the dataset, while I need to refresh the source. I looked at the source code and it seems that the method for refreshing the source (refreshFull in SourceMetadataManager) is not exposed and cannot be called through the restful api.
@bigfacewo That is correct but we are working on a feature to do on demand forget and/or refresh of all promoted datasets, will be available in the coming versions
It’s 2025 now, has the “refresh all” button come yet? I mean Hue, DBeaver, DataGrip,… anything has a “refresh” button to refresh schema for all tables from the source
At this point that feature has got deprioritized as product roadmap over the last 18 months has moved to Iceberg which no longer needs Metadata refresh
Thanks
Bali