ClassNotFoundException: Class com.dremio.exec.store.hive.GlueAWSCredentialsProvider not found

Pavlo · October 28, 2024, 9:43pm

Hello,
can you please help with the following issue?

When querying table from AWS Glue catalog Dremio fails with:

ClassNotFoundException: Class com.dremio.exec.store.hive.GlueAWSCredentialsProvider not found

I’m following the steps from the blog post: https://www.dremio.com/blog/bi-dashboards-with-apache-iceberg-using-aws-glue-and-apache-superset/#h-connecting-your-aws-glue-catalog-to-dremio

AWS Glue catalog was added successfully. AWS keys were provided for authentication.
Iceberg Tables are listed.
But when the SQL is executed the job fails with the ClassNotFoundException: Class com.dremio.exec.store.hive.GlueAWSCredentialsProvider not found error.

Here is the snippet from logs:

dremio-oss  | 2024-10-28 21:29:26,264 [FABRIC-rpc-event-queue] INFO  c.d.sabot.exec.FragmentExecutors - Received remote fragment start
instruction for 18dffdc8-f27b-812b-155f-fe7c8acc0800:0:0 with assigned weight 1 and scheduling weight 1
dremio-oss  | 2024-10-28 21:29:26,799 [e12 - 18dffdc8-f27b-812b-155f-fe7c8acc0800:frag:0:0] ERROR c.d.plugins.s3.store.S3FileSystem - E
rror while listing S3 buckets
dremio-oss  | java.util.concurrent.ExecutionException: java.io.IOException: From option fs.s3a.aws.credentials.provider java.lang.Class
NotFoundException: Class com.dremio.exec.store.hive.GlueAWSCredentialsProvider not found
dremio-oss  |   at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:594)
dremio-oss  |   at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:553)
dremio-oss  |   at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:110)
dremio-oss  |   at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:247)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2349)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079)
dremio-oss  |   at com.google.common.cache.LocalCache.get(LocalCache.java:4017)
dremio-oss  |   at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4040)
dremio-oss  |   at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4989)
dremio-oss  |   at com.dremio.plugins.s3.store.S3FileSystem.getS3V1Client(S3FileSystem.java:829)
dremio-oss  |   at com.dremio.plugins.s3.store.S3FileSystem.getContainerCreators(S3FileSystem.java:289)
dremio-oss  |   at com.dremio.plugins.util.ContainerFileSystem.refreshFileSystems(ContainerFileSystem.java:108)
dremio-oss  |   at com.dremio.plugins.util.ContainerFileSystem.initialize(ContainerFileSystem.java:185)
dremio-oss  |   at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3611)
dremio-oss  |   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:554)
dremio-oss  |   at com.dremio.exec.hadoop.HadoopFileSystem.get(HadoopFileSystem.java:123)
dremio-oss  |   at com.dremio.exec.store.hive.exec.DremioFileSystem.initialize(DremioFileSystem.java:114)
dremio-oss  |   at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3611)
dremio-oss  |   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:554)
dremio-oss  |   at com.dremio.exec.store.hive.exec.HadoopFsCacheWrapperPluginClassLoader$1.lambda$load$0(HadoopFsCacheWrapperPluginClas
sLoader.java:76)
dremio-oss  |   at java.base/java.security.AccessController.doPrivileged(Native Method)
dremio-oss  |   at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
dremio-oss  |   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
dremio-oss  |   at com.dremio.exec.store.hive.exec.HadoopFsCacheWrapperPluginClassLoader$1.load(HadoopFsCacheWrapperPluginClassLoader.j
ava:80)
dremio-oss  |   at com.dremio.exec.store.hive.exec.HadoopFsCacheWrapperPluginClassLoader$1.load(HadoopFsCacheWrapperPluginClassLoader.j
ava:60)
dremio-oss  |   at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3574)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2316)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079)
dremio-oss  |   at com.google.common.cache.LocalCache.get(LocalCache.java:4017)
dremio-oss  |   at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4040)
dremio-oss  |   at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4989)
dremio-oss  |   at com.dremio.exec.store.hive.exec.HadoopFsCacheWrapperPluginClassLoader.lambda$getHadoopFsSupplierPluginClassLoader$0(
HadoopFsCacheWrapperPluginClassLoader.java:96)
dremio-oss  |   at com.dremio.exec.store.hive.Hive3StoragePlugin.createFileSystem(Hive3StoragePlugin.java:419)
dremio-oss  |   at com.dremio.exec.store.hive.Hive3StoragePlugin.createFSWithAsyncOptions(Hive3StoragePlugin.java:456)
dremio-oss  |   at com.dremio.plugins.awsglue.store.AWSGlueStoragePlugin.createFSWithAsyncOptions(AWSGlueStoragePlugin.java:380)
dremio-oss  |   at com.dremio.exec.store.iceberg.IcebergManifestListScanCreator.create(IcebergManifestListScanCreator.java:47)
dremio-oss  |   at com.dremio.exec.store.iceberg.IcebergManifestListScanCreator.create(IcebergManifestListScanCreator.java:34)
dremio-oss  |   at com.dremio.sabot.driver.OperatorCreatorRegistry.getProducerOperator(OperatorCreatorRegistry.java:132)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan(PipelineCreator.java:241)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractSubScan.accept(AbstractSubScan.java:81)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitTableFunction(PipelineCreator.java:289)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitTableFunction(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.config.AbstractTableFunctionPOP.accept(AbstractTableFunctionPOP.java:46)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitTableFunction(PipelineCreator.java:289)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitTableFunction(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.config.AbstractTableFunctionPOP.accept(AbstractTableFunctionPOP.java:46)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:308)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitLimit(AbstractPhysicalVisitor.java:99)
dremio-oss  |   at com.dremio.exec.physical.config.Limit.accept(Limit.java:60)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:308)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitLimit(AbstractPhysicalVisitor.java:99)
dremio-oss  |   at com.dremio.exec.physical.config.Limit.accept(Limit.java:60)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:308)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject(AbstractPhysicalVisitor.java:84)
dremio-oss  |   at com.dremio.exec.physical.config.Project.accept(Project.java:54)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:308)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject(AbstractPhysicalVisitor.java:84)
dremio-oss  |   at com.dremio.exec.physical.config.Project.accept(Project.java:54)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:308)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriter(AbstractPhysicalVisitor.java:69)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractWriter.accept(AbstractWriter.java:30)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:308)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriterCommiter(AbstractPhysicalVisitor.java:59)
dremio-oss  |   at com.dremio.exec.physical.config.WriterCommitterPOP.accept(WriterCommitterPOP.java:119)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:308)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject(AbstractPhysicalVisitor.java:84)
dremio-oss  |   at com.dremio.exec.physical.config.Project.accept(Project.java:54)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitScreen(PipelineCreator.java:270)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitScreen(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.config.Screen.accept(Screen.java:60)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator.get(PipelineCreator.java:109)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator.get(PipelineCreator.java:103)
dremio-oss  |   at com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution(FragmentExecutor.java:724)
dremio-oss  |   at com.dremio.sabot.exec.fragment.FragmentExecutor.run(FragmentExecutor.java:514)
dremio-oss  |   at com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run(FragmentExecutor.java:1234)
dremio-oss  |   at com.dremio.sabot.task.AsyncTaskWrapper.run(AsyncTaskWrapper.java:130)
dremio-oss  |   at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:279)
dremio-oss  |   at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:186)
dremio-oss  | Caused by: java.io.IOException: From option fs.s3a.aws.credentials.provider java.lang.ClassNotFoundException: Class com.d
remio.exec.store.hive.GlueAWSCredentialsProvider not found
dremio-oss  |   at org.apache.hadoop.fs.s3a.S3AUtils.loadAWSProviderClasses(S3AUtils.java:611)
dremio-oss  |   at org.apache.hadoop.fs.s3a.S3AUtils.buildAWSProviderList(S3AUtils.java:634)
dremio-oss  |   at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:585)
dremio-oss  |   at com.dremio.plugins.s3.store.S3FileSystem.createS3V1Client(S3FileSystem.java:223)
dremio-oss  |   at com.dremio.plugins.s3.store.S3FileSystem$3.load(S3FileSystem.java:191)
dremio-oss  |   at com.dremio.plugins.s3.store.S3FileSystem$3.load(S3FileSystem.java:187)
dremio-oss  |   at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3574)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2316)
dremio-oss  |   ... 83 common frames omitted
dremio-oss  | Caused by: java.lang.ClassNotFoundException: Class com.dremio.exec.store.hive.GlueAWSCredentialsProvider not found
dremio-oss  |   at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2597)
dremio-oss  |   at org.apache.hadoop.conf.Configuration.getClasses(Configuration.java:2668)
dremio-oss  |   at org.apache.hadoop.fs.s3a.S3AUtils.loadAWSProviderClasses(S3AUtils.java:608)
dremio-oss  |   ... 90 common frames omitted
dremio-oss  | 2024-10-28 21:29:26,804 [e12 - 18dffdc8-f27b-812b-155f-fe7c8acc0800:frag:0:0] ERROR c.d.s.exec.fragment.FragmentExecutor
- ClassNotFoundException: Class com.dremio.exec.store.hive.GlueAWSCredentialsProvider not found
dremio-oss  | com.dremio.common.exceptions.UserException: ClassNotFoundException: Class com.dremio.exec.store.hive.GlueAWSCredentialsPr
ovider not found
dremio-oss  |   at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:984)
dremio-oss  |   at com.dremio.sabot.exec.fragment.FragmentExecutor.retire(FragmentExecutor.java:877)
dremio-oss  |   at com.dremio.sabot.exec.fragment.FragmentExecutor.finishRun(FragmentExecutor.java:768)
dremio-oss  |   at com.dremio.sabot.exec.fragment.FragmentExecutor.run(FragmentExecutor.java:604)
dremio-oss  |   at com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run(FragmentExecutor.java:1234)
dremio-oss  |   at com.dremio.sabot.task.AsyncTaskWrapper.run(AsyncTaskWrapper.java:130)
dremio-oss  |   at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:279)
dremio-oss  |   at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:186)
dremio-oss  | Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: java.io.IOException
: java.util.concurrent.ExecutionException: java.io.IOException: From option fs.s3a.aws.credentials.provider java.lang.ClassNotFoundExce
ption: Class com.dremio.exec.store.hive.GlueAWSCredentialsProvider not found
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2085)
dremio-oss  |   at com.google.common.cache.LocalCache.get(LocalCache.java:4017)
dremio-oss  |   at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4040)
dremio-oss  |   at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4989)
dremio-oss  |   at com.dremio.exec.store.hive.exec.HadoopFsCacheWrapperPluginClassLoader.lambda$getHadoopFsSupplierPluginClassLoader$0(
HadoopFsCacheWrapperPluginClassLoader.java:96)
dremio-oss  |   at com.dremio.exec.store.hive.Hive3StoragePlugin.createFileSystem(Hive3StoragePlugin.java:419)
dremio-oss  |   at com.dremio.exec.store.hive.Hive3StoragePlugin.createFSWithAsyncOptions(Hive3StoragePlugin.java:456)
dremio-oss  |   at com.dremio.plugins.awsglue.store.AWSGlueStoragePlugin.createFSWithAsyncOptions(AWSGlueStoragePlugin.java:380)
dremio-oss  |   at com.dremio.exec.store.iceberg.IcebergManifestListScanCreator.create(IcebergManifestListScanCreator.java:47)
dremio-oss  |   at com.dremio.exec.store.iceberg.IcebergManifestListScanCreator.create(IcebergManifestListScanCreator.java:34)
dremio-oss  |   at com.dremio.sabot.driver.OperatorCreatorRegistry.getProducerOperator(OperatorCreatorRegistry.java:132)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan(PipelineCreator.java:241)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractSubScan.accept(AbstractSubScan.java:81)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitTableFunction(PipelineCreator.java:289)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitTableFunction(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.config.AbstractTableFunctionPOP.accept(AbstractTableFunctionPOP.java:46)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitTableFunction(PipelineCreator.java:289)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitTableFunction(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.config.AbstractTableFunctionPOP.accept(AbstractTableFunctionPOP.java:46)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:308)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitLimit(AbstractPhysicalVisitor.java:99)
dremio-oss  |   at com.dremio.exec.physical.config.Limit.accept(Limit.java:60)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:308)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitLimit(AbstractPhysicalVisitor.java:99)
dremio-oss  |   at com.dremio.exec.physical.config.Limit.accept(Limit.java:60)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:308)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject(AbstractPhysicalVisitor.java:84)
dremio-oss  |   at com.dremio.exec.physical.config.Project.accept(Project.java:54)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:308)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject(AbstractPhysicalVisitor.java:84)
dremio-oss  |   at com.dremio.exec.physical.config.Project.accept(Project.java:54)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:308)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriter(AbstractPhysicalVisitor.java:69)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractWriter.accept(AbstractWriter.java:30)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:308)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriterCommiter(AbstractPhysicalVisitor.java:59)
dremio-oss  |   at com.dremio.exec.physical.config.WriterCommitterPOP.accept(WriterCommitterPOP.java:119)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:308)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject(AbstractPhysicalVisitor.java:84)
dremio-oss  |   at com.dremio.exec.physical.config.Project.accept(Project.java:54)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitScreen(PipelineCreator.java:270)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitScreen(PipelineCreator.java:123)
dremio-oss  |   at com.dremio.exec.physical.config.Screen.accept(Screen.java:60)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator.get(PipelineCreator.java:109)
dremio-oss  |   at com.dremio.sabot.driver.PipelineCreator.get(PipelineCreator.java:103)
dremio-oss  |   at com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution(FragmentExecutor.java:724)
dremio-oss  |   at com.dremio.sabot.exec.fragment.FragmentExecutor.run(FragmentExecutor.java:514)
dremio-oss  |   ... 4 common frames omitted
dremio-oss  | Caused by: java.lang.RuntimeException: java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException:
 From option fs.s3a.aws.credentials.provider java.lang.ClassNotFoundException: Class com.dremio.exec.store.hive.GlueAWSCredentialsProvi
der not found
dremio-oss  |   at com.dremio.exec.store.hive.exec.HadoopFsCacheWrapperPluginClassLoader$1.load(HadoopFsCacheWrapperPluginClassLoader.j
ava:82)
dremio-oss  |   at com.dremio.exec.store.hive.exec.HadoopFsCacheWrapperPluginClassLoader$1.load(HadoopFsCacheWrapperPluginClassLoader.j
ava:60)
dremio-oss  |   at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3574)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2316)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079)
dremio-oss  |   ... 58 common frames omitted
dremio-oss  | Caused by: java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: From option fs.s3a.aws.cred
entials.provider java.lang.ClassNotFoundException: Class com.dremio.exec.store.hive.GlueAWSCredentialsProvider not found
dremio-oss  |   at com.dremio.plugins.s3.store.S3FileSystem.getContainerCreators(S3FileSystem.java:311)
dremio-oss  |   at com.dremio.plugins.util.ContainerFileSystem.refreshFileSystems(ContainerFileSystem.java:108)
dremio-oss  |   at com.dremio.plugins.util.ContainerFileSystem.initialize(ContainerFileSystem.java:185)
dremio-oss  |   at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3611)
dremio-oss  |   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:554)
dremio-oss  |   at com.dremio.exec.hadoop.HadoopFileSystem.get(HadoopFileSystem.java:123)
dremio-oss  |   at com.dremio.exec.store.hive.exec.DremioFileSystem.initialize(DremioFileSystem.java:114)
dremio-oss  |   at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3611)
dremio-oss  |   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:554)
dremio-oss  |   at com.dremio.exec.store.hive.exec.HadoopFsCacheWrapperPluginClassLoader$1.lambda$load$0(HadoopFsCacheWrapperPluginClas
sLoader.java:76)
dremio-oss  |   at java.base/java.security.AccessController.doPrivileged(Native Method)
dremio-oss  |   at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
dremio-oss  |   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
dremio-oss  |   at com.dremio.exec.store.hive.exec.HadoopFsCacheWrapperPluginClassLoader$1.load(HadoopFsCacheWrapperPluginClassLoader.j
ava:80)
dremio-oss  |   ... 63 common frames omitted
dremio-oss  | Caused by: java.util.concurrent.ExecutionException: java.io.IOException: From option fs.s3a.aws.credentials.provider java
.lang.ClassNotFoundException: Class com.dremio.exec.store.hive.GlueAWSCredentialsProvider not found
dremio-oss  |   at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:594)
dremio-oss  |   at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:553)
dremio-oss  |   at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:110)
dremio-oss  |   at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:247)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2349)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079)
dremio-oss  |   at com.google.common.cache.LocalCache.get(LocalCache.java:4017)
dremio-oss  |   at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4040)
dremio-oss  |   at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4989)
dremio-oss  |   at com.dremio.plugins.s3.store.S3FileSystem.getS3V1Client(S3FileSystem.java:829)
dremio-oss  |   at com.dremio.plugins.s3.store.S3FileSystem.getContainerCreators(S3FileSystem.java:289)
dremio-oss  |   ... 76 common frames omitted
dremio-oss  | Caused by: java.io.IOException: From option fs.s3a.aws.credentials.provider java.lang.ClassNotFoundException: Class com.d
remio.exec.store.hive.GlueAWSCredentialsProvider not found
dremio-oss  |   at org.apache.hadoop.fs.s3a.S3AUtils.loadAWSProviderClasses(S3AUtils.java:611)
dremio-oss  |   at org.apache.hadoop.fs.s3a.S3AUtils.buildAWSProviderList(S3AUtils.java:634)
dremio-oss  |   at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:585)
dremio-oss  |   at com.dremio.plugins.s3.store.S3FileSystem.createS3V1Client(S3FileSystem.java:223)
dremio-oss  |   at com.dremio.plugins.s3.store.S3FileSystem$3.load(S3FileSystem.java:191)
dremio-oss  |   at com.dremio.plugins.s3.store.S3FileSystem$3.load(S3FileSystem.java:187)
dremio-oss  |   at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3574)
dremio-oss  |   at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2316)
dremio-oss  |   ... 83 common frames omitted
dremio-oss  | Caused by: java.lang.ClassNotFoundException: Class com.dremio.exec.store.hive.GlueAWSCredentialsProvider not found
dremio-oss  |   at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2597)
dremio-oss  |   at org.apache.hadoop.conf.Configuration.getClasses(Configuration.java:2668)
dremio-oss  |   at org.apache.hadoop.fs.s3a.S3AUtils.loadAWSProviderClasses(S3AUtils.java:608)
dremio-oss  |   ... 90 common frames omitted
dremio-oss  | 2024-10-28 21:29:26,820 [Fabric-RPC-Offload12] INFO  c.d.exec.maestro.FragmentTracker - Fragment 18dffdc8-f27b-812b-155f-
fe7c8acc0800:0:0 failed, cancelling remaining fragments.
dremio-oss  | 2024-10-28 21:29:26,823 [Fabric-RPC-Offload12] INFO  c.d.exec.work.foreman.AttemptManager - 18dffdc8-f27b-812b-155f-fe7c8
acc0800: State change requested RUNNING --> FAILED, Exception com.dremio.common.exceptions.UserRemoteException: SYSTEM ERROR: ClassNotF
oundException: Class com.dremio.exec.store.hive.GlueAWSCredentialsProvider not found

balaji.ramaswamy · November 1, 2024, 12:13am

@Pavlo To just rule out that OSS build is not the issue, can you please download the community version and try the same?

neriberto · November 1, 2024, 7:26pm

I’m encountering the same error. I’m running Docker on an EC2 instance to test, and I’ve tried both dremio/dremio-oss:25.1.1 and dremio/dremio-oss:latest

Reading directly from S3 works, but accessing data from the Glue Catalog does not.

balaji.ramaswamy · November 7, 2024, 6:29am

@neriberto Does it work on Community edition instead of OSS? Reason I am asking is that I see a Class com.d remio.exec.store.hive.GlueAWSCredentialsProvider not found

Topic		Replies	Views
Unable to connect to Glue Data Catalog on Dremio 22.0.0	0	1127	July 14, 2022
Problems with connection to AWS Glue Catalog Iceberg table Dremio University	4	1688	September 13, 2022
Unable to access iceberg data created through AWS glue and stored in s3 Dremio Cloud	7	891	September 14, 2023
Connect Dremio to Iceberg Hive Catalog located at S3 Dremio University	2	1139	August 15, 2023
Iceberg Dataset not found	3	1366	August 17, 2022

ClassNotFoundException: Class com.dremio.exec.store.hive.GlueAWSCredentialsProvider not found

Related topics