Unable to promote Iceberg hadoop dataset with all metadata files in gzip format

Dremio doesn’t seem to recognize a table where all metadata files where written with write.metadata.compression-codec=gzip

If I unzip and upload at least one metadata file without the .gz in the name, the table can be promoted.

Edit: This is only an issue when promoting through the Dremio UI. When I promote it via the REST API, it works even with all metadata files compressed

@dotjdk This is currently not supported

It is supported as long as there is at least one file in the folder that matches the non-gzipped metadata naming structure. It must just be an issue of pattern matching in the UI.

It is also supported if the iceberg table is promoted via the REST API. Even though all metadata files are with the gzipped naming structure.

Everything works for us after I figured out the kink. We actually usually promote our datasets via REST, so it hasn’t really been an issue. My message was just meant as a courteous bug report.

Hi @dotjdk

When I say not supported, this is something that is not part of the testing and probably a future planned release. That is why it does not work out of the box and you had to make it work by keeping one folder unzipped

Totally appreciate your help in testing this and providing feedback

Thanks again

1 Like