Iceberg support GA

1

Hi Team,

Iceberg support GA in Dremio?

How to create iceberg table in Dremio?

Thanks,
KP

1 Like
2

@kvpr949 Currently in 18.0 Dremio uses Iceberg format for it’s metadata storage and retrieval, GA for file format support will be next

3

Thank you Balaji.

So I can’t you iceberg in 18.0?

Thanks,
KP

4

@kvpr949 Feature is not fully GA in 18.0 so if you can wait, it would be great

5

Hello Balaji,
Now I’m use version 18 community. I can’t read the iceberg table. I can query data on spark or trino. How can i access data via Dremio

Thanks

image
image1600×900 309 KB

6

@Haodochi Is your Iceberg table via Hadoop catalog or Hive catalog?

8

Hi Balaji,
My Iceberg table now in Hadoop

Thanks

9

In last time i use the version 15, it’s can read the Iceberg table

10

Hi @Haodochi In 18.0 enable dremio.iceberg.enabled and try, In 18.0 it is still la preview feature and will be GA soon

11

Hi, is there any update on this? I just looked at the source and haven’t seen much implementation for iceberg yet. Can iceberg become GA in 19.0 release?

12

@chulucninh09 At this point, IceBerg GA is targeted for 20.0, please see release note of 19.0 coming out soon and it will have an mention in case it is GA in 19.0

1 Like
13

Hi @balaji.ramaswamy, there is a comment in changelog mentioned iceberg support. Can you clarify this a bit more.
https://docs.dremio.com/release-notes/1900-release-notes/

Apache Iceberg Hive Table Support
Dremio 19.0+ supports using the popular Apache Iceberg open table format created with Spark 3. This functionality extends to users with read (SELECT) privileges on sources that support Hive catalogs (ADLS, Hive, S3). Additionally, this allows for more rapid storage and retrieval of metadata when after queries or changes are made to a dataset.

As I understand, if iceberg use catalog in spark like this:

--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
    --conf spark.sql.catalog.spark_catalog.type=hive

Then I can use dremio to read iceberg dataset, is that right?

14

Don’t worry about my question, Dremio has supported iceberg just enough for me right now.

Following the docs here: Dremio

No need for hive catalog, I set iceberg.catalog_type=hive in S3 and things went well.

image
image1188×724 38.6 KB

15

@chulucninh09 Dremio 19.0 has Hive catalog support for Apache iceberg

https://docs.dremio.com/release-notes/1900-release-notes/#apache-iceberg-hive-table-support

1 Like