How can we automatically format Parquet files in Dremio that are connected via MinIO? We have already done it manually and enabled the “Automatically Format” option. Are there any other ways to automate this process?
If you’ve already enabled “Automatically format files into tables when users issue queries”, that’s the primary built-in automation for Parquet files in Dremio (including MinIO/S3 sources).
With this enabled, Dremio will automatically promote new Parquet files to physical datasets the first time they are queried — no manual formatting is needed. This is the recommended approach for dynamic environments where files are added frequently.
Hi, thanks for replying
Is there any other way to do it
and can you please help me to understand how it happens in production level (especially with source formatting)
@prudhvi You can use the API to do this