Can't access Iceberg tables

Dremio Cloud
1

I have a Dremio account setup in AWS and I am trying to query Iceberg tables saved in s3.
Before I can promote the tables I have to set the following in the support console:
ALTER SYSTEM SET dremio.iceberg.enabled = true;
ALTER SYSTEM SET reflection.manager.use_iceberg_dataset.enabled = true;

I however can’t find the support console as it’s not available in my account. How can I find the support console?

2

Hi Cano,

As of now, Dremio Cloud users do not have access to change support keys. Can you please attach the profile for the failed job?

dremio.iceberg.enabled should be true by default

3

Hi Hope,

Thanks for the response. I haven’t been able to query the Iceberg tables. When I try to promote it from Dataset settings you can see that it automatically detects that it’s Iceberg format. But once I click on “save” I get an error message saying “Failed to get iceberg metadata”.

I would really appreciate a workaround for this if there’s any, thank you!

4

This error is noted as a known issue here. Workaround is:

  1. Use ALTER PDS to forget the metadata for affected datasets (see Forgetting Physical Dataset Metadata).
  2. Use ALTER PDS to refresh the metadata for affected datasets (see Refreshing Physical Dataset Metadata).

Please let me know if that helps!

1 Like
5

I have same error, cannot forget metadata because is not yet a PDS so error if try forget metadata
this happend when we have folder structure for example

  • credito/2022/07/15/dataset
  • credito/2022/07/16/dataset

this error is only in new versions because use iceberg older version use parquet and not have this problem

6

@dacopan and @Cano

Which Iceberg catalog are you using?

7

Hello, thank you your reply, I’m creating child tables with cta in a folder year/month/day estructure, you can view more details in my post, also error from logs

8

Hi Hope, the version of Iceberg I’m using is 2.0, but it was created using ORC. I’m assuming Parquet is the default file format for Iceberg tables in Dremio?

9

Dremio Cloud only supports Iceberg files directly if they use the HadoopCatalog. If it’s a different iceberg catalog, you will need to add the catalog as a source instead (such as with Glue here).