Iceberg file size on dremio

Dremio Cloud
1

Hi,

We have generated parquet file of size 1GiB and uploaded on azure storage.
Approximate data on this file is around 8460000 records with 30 columns.

After moving this data to iceberg table, I could see 20 files are generated of 256MB.

I understand the size of file is default to 256 mb on dremio, but the volume of data after moving to iceberg is 5x.

@balaji.ramaswamy : Am I missing any compression config. Need some help on this.

Thanks
Ajay

2

@ajay098 is data+metadata 5 GB?

3

Yes. File has 10 days of data and iceberg table is partitioned on day.

parquet file

image
image902×70 12.7 KB

After creating iceberg table

image
image902×134 26.5 KB

image
image902×454 99.7 KB

4

@ajay098 Validating, will get back to you, please ping if you do not hear back soon

5

@ajay098 How did you create the Iceberg tables? Via Dremio? or Via an external tool? If via Dremio, are you able to send us the profile for the DDL/DML job?

6

Hi,

Unfortunately job has retention of 7 days.

We created table via Dremio.

Files were created with following info.
library version 1.5.0
parquet-avro version 1.13.1

Parquet version V1
Compression Technique gzip
Encoding PLAIN_DICTIONARY

Thanks
Ajay