I think a standard pattern is to use some “as-atomic-as-possible” operation to write parquet files and then update a reference.
When you point Dremio at a parquet dir that gets updated, the pattern is broken until Dremio refreshes.
What is the right pattern here?