I have mapped a NAS drive on dremio and formatted the drive into tabular format to query via Dremio.
This drive contains a csv file, which is updated every four hours(file name remains unchanged) - I was expecting that dremio would read the most latest file from the mapped drive (Containing updated CSV files).
I see variation in count of records in dremio vs the actual updated file.
Please note I have tried with and without reflections(hoping this could be related to caching of stale data) - in both cases I see that dremio does not read the latest csv file.
Assuming you promoted a folder containing csv files, then as long as you are only changing previously existing files things should work as you expect - running a count(*) should always return the correct result. Note that preview queries may cache results so make sure its a full run.
If you are adding new files under the folder, these new files won’t be immediately detected - sources in Dremio have a metadata policy (edit a source and click on the metadata section). The default metadata refresh is one hour.
Reflections will indeed only show data that was present when the reflection was built - so that would depend on how often your reflections are refreshed (default is again one hour).
we are dropping the file via script which was only changing the last update time of the file and not the directory. using a touch command we modified the last update time of the directory and it worked.