Iceberg file size on dremio

[Ken]

Hi @dacropan,

I have the same observation that the table files tracked by Dremio (via the select * from table(table_files('XXX')) sql where XXX is the fully qualified path of the table) is far fewer than the parquet data files under the iceberg folder on Disk.

My setup is Dremio with Minio as the object storage.

My thinking is that these are orphan data files which Dremio fails to delete.

My solution is to run a list object function in Minio and delete those which are not tracked by Dremio. Below is my solution for your reference.

Create Table NOT as iceberg

To summarise, this approach works to delete data files not longer tracked by Dremio, but still remains in Minio. Step 1. Query the data_files in Dremio select "file_path" from table( table_files('XXXX')) This list represents the list of data files tracked by Dremio. Step 2. Use the list_objects function in Minio Client to list all the objects under the iceberg table folder objects = self.client.list_objects( bucket_name, prefix=bucket_subpath, recursive…