Hi @balaji.ramaswamy /@doron /All,
We have implemented “METADATA REFRESH” for every minute to ADLS Datasource in Dremio Ver - 19.0.0-20211027034112059. The refreshes are running fine but we could see its executes on folders randomly from source datasource which causing real time refresh delay for current month/day folders, For ex. in todays run it refreshing October 2021 data which is not required. This creates to utilize more Dremio resource which impacts the normal queries to slow down in results.
Kindly suggest us the solution to enable metadata refresh only for current month/day folder.
Dremio Config Details as follows.
1 Master Node
1 Coordinator Node
3 Executors Node
Thank You!
WR,
Sathish R