Connect to Nessie catalog

I have a couple of iceberg tables in S3 and configured via the Nessie catalog.

How can I connect to a Nessie catalog and query the iceberg tables ?

Is it only possible via directly connecting to S3 and reading the iceberg directly?

Dremio Version Info

Build : 23.1.0-202211250136090978-a79618c7
Edition : Community Edition
Build Time: 11/25/2022 01:55:10

@sarathmaf Via the Dremio UI, are you able to add the S3 source and promote the Iceberg dataset and then query?

@sarathmaf It sounds like you setup a Nessie server and created a few Iceberg tables via Spark? Adding that Nessie catalog as a source into Dremio Software is on the roadmap. What Balaji suggested allows you to directly query that Iceberg table from Dremio but ideally this should be done by first getting the root metadata pointer from Nessie.

BTW, when you do a CTAS in Dremio and create an Iceberg table, we use the following catalogs depending on which source you wrote to: Dremio

Yes… now if we try to pint to the file of the data or metadata it is not possible. Or can we manually get the root meta data and use this as the S3 location?

No … iceberg created via Nessie has data and metadata folders. So in data it has the part parquet files and metadata folder hs the meta info as json.

So in this case which one we will point as data ?

I guess it doesn’t work (which is a good thing). You can’t directly query the Iceberg tables managed by Nessie catalog.

@sarathmaf I think your only option right now if you want to use Nessie with Dremio Sonar query engine is through Dremio Cloud and Arctic.

1 Like