Add_files and schema.name-mapping.default

Does Dremio support parquet files added to Iceberg table with add_files procedure?

https://iceberg.apache.org/docs/1.5.2/spark-procedures/#add_files

After execution of this procedure metadata files have been added, and run of COUNT(*) in Dremio shows the correct number of records. But all values are displayed as NULL. Why does Dremio can’t read values from these new files?

Parquet files added with this procedure don’t have field_id inside parquet schema. But this metadata is used in Iceberg tables. To overcome this issue Iceberg tables can have a property schema.name-mapping.default Spec - Apache Iceberg which should be used when parquet files have no field_id

Does Dremio use schema.name-mapping.default? If Dremio takes this property into consideration, why all values are displayed as NULL?

Note: added parquet files with field_id defined in the parquet schema are being queried by Dremio with no issue.

@enkeipy Let me check on this and get back to you

1 Like