We have a folder with parquet files, and created a “dataset” in Dremio from it. Folders appear as dir0, dir1, dir2… And so on, dir0 is data partitioned by year, dir1 by month, and dir2 by day. Every day, a new folder within dir2 is created, with the data of that day. At first sight, it seems Dremio detects these new folders, and we are able to run queries on them. However, at certain point, all new folders don’t appear in the dataset anymore. Is there anyway to force Dremio to refresh the new data in a dataset? Please note that we are not talking about reflections, but just the folder with parquet files where we want to run queries
There is a background refresh job that by defaults does it every hour. Open the source, click on metadata tab and see if that has changed, alternatively you can force it when you know the dataset in the backend has changed