We are currently testing the new functionality released in dremio 18 around real time metadata refresh. I was trying to test it, but it does not seem to work as expected (maybe my expectation is wrong).
Both flags are enabled
I have an s3 data lake source connected to local minio instance containing parquet files. My expectation would be, if I add a new file to the folder that already has been promoted as a physical data set, I do not need to refresh its metadata and see new data almost instantly. however this is not the case.
Is my expectation wrong? Or is it some limitation in minio?