I created a data sou3rce as Hive 3.x metastore to read tables stored in S3 compatible storage with hive catalog in iceberg table format. The following properties are defined in the advanced options:
fs.s3a.access.key= <access.key>
fs.s3a.secret.key=<secret.key>
fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
fs.s3a.endpoint=
fs.s3a.path.style.access=true
hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
I can brows the list of databases and tables. But when I try to select from any table I get the following error:
s3a://iceberg-test3/data/metadata/00000-68642976-ae50-4da4-8a64-9ed8a5aa3a58.metadata.json: getFileStatus on s3a://iceberg-test3/data/metadata/00000-68642976-ae50-4da4-8a64-9ed8a5aa3a58.metadata.json: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 0a8b31d2:182c7f08b15:a4bcf:52; S3 Extended Request ID: ; Proxy: null), S3 Extended Request ID: :403 Forbidden.
The access key has full permissions to the bucket.
Any idea what’s happening? Please note that I can select from the same catalog and tables which I created from spark-sql.
Access key and secret key were switched! I am getting file not found now while the file already exists…
Here is the Exception I am getting now…
2022-12-08 20:26:47,232 [qtp827206475-207] WARN org.apache.iceberg.util.Tasks - Retrying task after failure: Failed to open input stream for file: s3a://iceberg-test3/data/mytable/metadata/00001-ddecd9af-e5cd-404c-905a-dbb5492967e9.metadata.json
org.apache.iceberg.exceptions.NotFoundException: Failed to open input stream for file: s3a://iceberg-test3/data/mytable/metadata/00001-ddecd9af-e5cd-404c-905a-dbb5492967e9.metadata.json
at org.apache.iceberg.hadoop.HadoopInputFile.newStream(HadoopInputFile.java:177)
at com.dremio.exec.store.iceberg.DremioInputFile.newStream(DremioInputFile.java:100)
at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:250)
at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:245)
at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$0(BaseMetastoreTableOperations.java:171)
at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$1(BaseMetastoreTableOperations.java:185)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190)
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:185)
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:170)
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:165)
at com.dremio.exec.store.hive.iceberg.IcebergHiveTableOperations.doRefresh(IcebergHiveTableOperations.java:44)
at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:95)
at org.apache.iceberg.BaseTable.refresh(BaseTable.java:59)
at com.dremio.exec.store.hive.metadata.HiveMetadataUtils.getTableMetadataFromIceberg(HiveMetadataUtils.java:561)
at com.dremio.exec.store.hive.metadata.HiveMetadataUtils.getTableMetadata(HiveMetadataUtils.java:533)
at com.dremio.exec.store.hive.Hive3StoragePlugin.listPartitionChunks(Hive3StoragePlugin.java:1295)
at com.dremio.exec.catalog.DatasetSaverImpl.saveUsingV1Flow(DatasetSaverImpl.java:248)
at com.dremio.exec.catalog.DatasetSaverImpl.save(DatasetSaverImpl.java:121)
at com.dremio.exec.catalog.DatasetSaverImpl.save(DatasetSaverImpl.java:143)
at com.dremio.exec.catalog.DatasetManager.getTableFromPlugin(DatasetManager.java:373)
at com.dremio.exec.catalog.DatasetManager.getTable(DatasetManager.java:215)
at com.dremio.exec.catalog.CatalogImpl.getTableHelper(CatalogImpl.java:472)
at com.dremio.exec.catalog.CatalogImpl.getTable(CatalogImpl.java:225)
at com.dremio.exec.catalog.SourceAccessChecker.lambda$getTable$3(SourceAccessChecker.java:128)
at com.dremio.exec.catalog.SourceAccessChecker.getIfVisible(SourceAccessChecker.java:97)
at com.dremio.exec.catalog.SourceAccessChecker.getTable(SourceAccessChecker.java:128)
at com.dremio.exec.catalog.DelegatingCatalog.getTable(DelegatingCatalog.java:105)
at com.dremio.exec.catalog.CachingCatalog.getTable(CachingCatalog.java:94)
at com.dremio.dac.explore.DatasetsResource.getDatasetSummary(DatasetsResource.java:314)
at com.dremio.dac.explore.DatasetsResource.newUntitled(DatasetsResource.java:152)
at com.dremio.dac.explore.DatasetsResource.newUntitledFromParent(DatasetsResource.java:247)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
Caused by: java.io.FileNotFoundException: No such file or directory: s3a://iceberg-test3/data/mytable/metadata/00001-ddecd9af-e5cd-404c-905a-dbb5492967e9.metadata.json
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3864)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3686)
at org.apache.hadoop.fs.s3a.S3AFileSystem.extractOrFetchSimpleFileStatus(S3AFileSystem.java:5399)
at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:1468)
at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:1444)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:976)
at org.apache.iceberg.hadoop.HadoopInputFile.newStream(HadoopInputFile.java:175)
... 94 common frames omitted