I use a folder in an Azure Blob storage to list multiple Excel files with the same structure.
I have configured this directory as a dataset in Dremio but Dremio never detects new files in this folder when they appear.
I have to manually force the refresh of metadata with this command :
ALTER PDS REFRESH METADATA FORCE UPDATE
Is it normal ?
Dremio only discovers new files when metadata is refresh (defaults to each hour) and can be configured in the UI:
You can also use that SQL command to force a refresh. Getting metadata can be expensive so Dremio doesn’t do it for each query.
I tried configuring the Dataset Discovery and Dataset Details refresh rates at “every 1 minute” but it does not discover my new files.
It is as if the normal metadata refresh job did not detect any changes in the metadata and then did nothing.
If I run ALTER PDS REFRESH METADATA, Dremio answers that nothing needs to be updated.
I need to add FORCE UPDATE.
