Near real time metadata refresh

Hello,

We are currently testing the new functionality released in dremio 18 around real time metadata refresh. I was trying to test it, but it does not seem to work as expected (maybe my expectation is wrong).

Both flags are enabled image

I have an s3 data lake source connected to local minio instance containing parquet files. My expectation would be, if I add a new file to the folder that already has been promoted as a physical data set, I do not need to refresh its metadata and see new data almost instantly. however this is not the case.

Is my expectation wrong? Or is it some limitation in minio?

@igorsechyn After how long do you see the data from the new file?

@balaji.ramaswamy just tested it in an aws environment with s3 bucket:

  • Delete a file from a folder promoted to a pds in dremio
  • immediately run a query on the pds

I can see how before running a query, metadata is being updated without me doing anything. Will test with adding a new file with new data, but i expect it will work. I suppose there is some incompatibility with minio