Hi all
I have this exception when we run the query to our S3 bucket, where we have each tenant’s parquets.
We can create the data source and connect to S3 without a problem; however, once the query runs, we get the exception below.
We have 1 coordinator node and 2 workers
If some one could guide me to fix this issue , I will really glad
Thanks
Pablo
VALIDATION ERROR: Failure getting source [tenant_0007].
ErrorOrigin: EXECUTOR
[Error Id: 9ea8842b-1641-43c6-bd8f-e13777034bf4 ]
(java.util.concurrent.ExecutionException) java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider instantiation exception: java.lang.RuntimeException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
java.util.concurrent.CompletableFuture.reportGet():395
java.util.concurrent.CompletableFuture.get():2022
com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin():1563
com.dremio.exec.catalog.ManagedStoragePlugin.replacePluginWithLock():1498
com.dremio.exec.catalog.PluginsManager.getSynchronized():557
com.dremio.exec.catalog.CatalogServiceImpl.getPlugin():670
com.dremio.exec.catalog.CatalogServiceImpl.getSource():731
com.dremio.sabot.exec.fragment.FragmentExecutionContext.getStoragePlugin():77
com.dremio.exec.store.parquet.ParquetSplitReaderCreatorIterator.():194
com.dremio.exec.store.parquet.ParquetSplitReaderCreatorIterator.():156
com.dremio.exec.store.parquet.ParquetOperatorCreator.create():42
com.dremio.exec.store.parquet.ParquetOperatorCreator.create():29
com.dremio.sabot.driver.OperatorCreatorRegistry.getProducerOperator():132
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan():241
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan():123
com.dremio.exec.physical.base.AbstractSubScan.accept():81
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():308
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():123
com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject():84
com.dremio.exec.physical.config.Project.accept():54
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():308
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():123
com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject():84
com.dremio.exec.physical.config.Project.accept():54
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():308
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():123
com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriter():69
com.dremio.exec.physical.base.AbstractWriter.accept():30
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSender():205
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSender():123
com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitSingleSender():201
com.dremio.exec.physical.config.SingleSender.accept():77
com.dremio.sabot.driver.PipelineCreator.get():109
com.dremio.sabot.driver.PipelineCreator.get():103
com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():724
com.dremio.sabot.exec.fragment.FragmentExecutor.run():514
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():1234
com.dremio.sabot.task.AsyncTaskWrapper.run():130
com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():279
com.dremio.sabot.task.slicing.SlicingThread.run():186
Caused By (java.io.IOException) java.util.concurrent.ExecutionException: java.io.IOException: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider instantiation exception: java.lang.RuntimeException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
com.dremio.plugins.s3.store.S3FileSystem.getContainerCreators():311
com.dremio.plugins.util.ContainerFileSystem.refreshFileSystems():108
com.dremio.plugins.util.ContainerFileSystem.initialize():185
com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():296
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():423
org.apache.hadoop.security.UserGroupInformation.doAs():1899
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():301
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():272
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574
com.google.common.cache.LocalCache$Segment.loadSync():2316
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189
com.google.common.cache.LocalCache$Segment.get():2079
com.google.common.cache.LocalCache.get():4017
com.google.common.cache.LocalCache.getOrLoad():4040
com.google.common.cache.LocalCache$LocalLoadingCache.get():4989
com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():555
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():538
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():530
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():508
com.dremio.exec.store.dfs.FileSystemPlugin.start():965
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$4():787
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$6():865
java.util.concurrent.CompletableFuture$AsyncSupply.run():1700
java.util.concurrent.ThreadPoolExecutor.runWorker():1128
java.util.concurrent.ThreadPoolExecutor$Worker.run():628
java.lang.Thread.run():829
Caused By (java.util.concurrent.ExecutionException) java.io.IOException: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider instantiation exception: java.lang.RuntimeException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
com.google.common.util.concurrent.AbstractFuture.getDoneValue():594
com.google.common.util.concurrent.AbstractFuture.get():553
com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get():110
com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly():247
com.google.common.cache.LocalCache$Segment.getAndRecordStats():2349
com.google.common.cache.LocalCache$Segment.loadSync():2317
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189
com.google.common.cache.LocalCache$Segment.get():2079
com.google.common.cache.LocalCache.get():4017
com.google.common.cache.LocalCache.getOrLoad():4040
com.google.common.cache.LocalCache$LocalLoadingCache.get():4989
com.dremio.plugins.s3.store.S3FileSystem.getS3V1Client():838
com.dremio.plugins.s3.store.S3FileSystem.getContainerCreators():289
com.dremio.plugins.util.ContainerFileSystem.refreshFileSystems():108
com.dremio.plugins.util.ContainerFileSystem.initialize():185
com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():296
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():423
org.apache.hadoop.security.UserGroupInformation.doAs():1899
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():301
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():272
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574
com.google.common.cache.LocalCache$Segment.loadSync():2316
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189
com.google.common.cache.LocalCache$Segment.get():2079
com.google.common.cache.LocalCache.get():4017
com.google.common.cache.LocalCache.getOrLoad():4040
com.google.common.cache.LocalCache$LocalLoadingCache.get():4989
com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():555
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():538
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():530
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():508
com.dremio.exec.store.dfs.FileSystemPlugin.start():965
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$4():787
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$6():865
java.util.concurrent.CompletableFuture$AsyncSupply.run():1700
java.util.concurrent.ThreadPoolExecutor.runWorker():1128
java.util.concurrent.ThreadPoolExecutor$Worker.run():628
java.lang.Thread.run():829
Caused By (java.io.IOException) org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider instantiation exception: java.lang.RuntimeException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider():749
org.apache.hadoop.fs.s3a.S3AUtils.buildAWSProviderList():659
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet():585
com.dremio.plugins.s3.store.S3FileSystem.createS3V1Client():223
com.dremio.plugins.s3.store.S3FileSystem$3.load():191
com.dremio.plugins.s3.store.S3FileSystem$3.load():187
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574
com.google.common.cache.LocalCache$Segment.loadSync():2316
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189
com.google.common.cache.LocalCache$Segment.get():2079
com.google.common.cache.LocalCache.get():4017
com.google.common.cache.LocalCache.getOrLoad():4040
com.google.common.cache.LocalCache$LocalLoadingCache.get():4989
com.dremio.plugins.s3.store.S3FileSystem.getS3V1Client():838
com.dremio.plugins.s3.store.S3FileSystem.getContainerCreators():289
com.dremio.plugins.util.ContainerFileSystem.refreshFileSystems():108
com.dremio.plugins.util.ContainerFileSystem.initialize():185
com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():296
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():423
org.apache.hadoop.security.UserGroupInformation.doAs():1899
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():301
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():272
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574
com.google.common.cache.LocalCache$Segment.loadSync():2316
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189
com.google.common.cache.LocalCache$Segment.get():2079
com.google.common.cache.LocalCache.get():4017
com.google.common.cache.LocalCache.getOrLoad():4040
com.google.common.cache.LocalCache$LocalLoadingCache.get():4989
com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():555
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():538
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():530
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():508
com.dremio.exec.store.dfs.FileSystemPlugin.start():965
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$4():787
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$6():865
java.util.concurrent.CompletableFuture$AsyncSupply.run():1700
java.util.concurrent.ThreadPoolExecutor.runWorker():1128
java.util.concurrent.ThreadPoolExecutor$Worker.run():628
java.lang.Thread.run():829
Caused By (java.lang.RuntimeException) io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
com.dremio.services.credentials.CredentialsProviderContextImpl.lookupOnLeader():68
com.dremio.services.credentials.SystemSecretCredentialsProvider.lambda$doLookup$0():78
com.dremio.telemetry.api.metrics.TimerUtils.timedExceptionThrowingOperation():124
com.dremio.services.credentials.SystemSecretCredentialsProvider.doLookup():75
com.dremio.services.credentials.AbstractSimpleCredentialsProvider.lookup():41
com.dremio.services.credentials.CredentialsServiceImpl.lookup():95
com.dremio.hadoop.security.alias.DremioCredentialProvider.lambda$resolveCredentialPattern$0():109
com.dremio.telemetry.api.metrics.TimerUtils.timedExceptionThrowingOperation():124
com.dremio.hadoop.security.alias.DremioCredentialProvider.resolveCredentialPattern():103
com.dremio.hadoop.security.alias.DremioCredentialProvider.getCredentialEntry():66
org.apache.hadoop.conf.Configuration.getCredentialEntry():2397
org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders():2444
org.apache.hadoop.conf.Configuration.getPassword():2377
org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword():933
org.apache.hadoop.fs.s3a.S3AUtils.getPassword():918
org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword():884
org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword():842
org.apache.hadoop.fs.s3a.S3AUtils.getAWSAccessKeys():800
org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider.():66
jdk.internal.reflect.GeneratedConstructorAccessor79.newInstance():-1
jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance():45
java.lang.reflect.Constructor.newInstance():490
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider():706
org.apache.hadoop.fs.s3a.S3AUtils.buildAWSProviderList():659
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet():585
com.dremio.plugins.s3.store.S3FileSystem.createS3V1Client():223
com.dremio.plugins.s3.store.S3FileSystem$3.load():191
com.dremio.plugins.s3.store.S3FileSystem$3.load():187
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574
com.google.common.cache.LocalCache$Segment.loadSync():2316
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189
com.google.common.cache.LocalCache$Segment.get():2079
com.google.common.cache.LocalCache.get():4017
com.google.common.cache.LocalCache.getOrLoad():4040
com.google.common.cache.LocalCache$LocalLoadingCache.get():4989
com.dremio.plugins.s3.store.S3FileSystem.getS3V1Client():838
com.dremio.plugins.s3.store.S3FileSystem.getContainerCreators():289
com.dremio.plugins.util.ContainerFileSystem.refreshFileSystems():108
com.dremio.plugins.util.ContainerFileSystem.initialize():185
com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():296
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():423
org.apache.hadoop.security.UserGroupInformation.doAs():1899
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():301
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():272
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574
com.google.common.cache.LocalCache$Segment.loadSync():2316
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189
com.google.common.cache.LocalCache$Segment.get():2079
com.google.common.cache.LocalCache.get():4017
com.google.common.cache.LocalCache.getOrLoad():4040
com.google.common.cache.LocalCache$LocalLoadingCache.get():4989
com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():555
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():538
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():530
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():508
com.dremio.exec.store.dfs.FileSystemPlugin.start():965
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$4():787
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$6():865
java.util.concurrent.CompletableFuture$AsyncSupply.run():1700
java.util.concurrent.ThreadPoolExecutor.runWorker():1128
java.util.concurrent.ThreadPoolExecutor$Worker.run():628
java.lang.Thread.run():829
Caused By (io.grpc.StatusRuntimeException) UNAVAILABLE: io exception
io.grpc.stub.ClientCalls.toStatusRuntimeException():268
io.grpc.stub.ClientCalls.getUnchecked():249
io.grpc.stub.ClientCalls.blockingUnaryCall():167
com.dremio.services.credentials.proto.CredentialsServiceGrpc$CredentialsServiceBlockingStub.lookup():160
com.dremio.services.credentials.CredentialsProviderContextImpl.lookupOnLeader():58
com.dremio.services.credentials.SystemSecretCredentialsProvider.lambda$doLookup$0():78
com.dremio.telemetry.api.metrics.TimerUtils.timedExceptionThrowingOperation():124
com.dremio.services.credentials.SystemSecretCredentialsProvider.doLookup():75
com.dremio.services.credentials.AbstractSimpleCredentialsProvider.lookup():41
com.dremio.services.credentials.CredentialsServiceImpl.lookup():95
com.dremio.hadoop.security.alias.DremioCredentialProvider.lambda$resolveCredentialPattern$0():109
com.dremio.telemetry.api.metrics.TimerUtils.timedExceptionThrowingOperation():124
com.dremio.hadoop.security.alias.DremioCredentialProvider.resolveCredentialPattern():103
com.dremio.hadoop.security.alias.DremioCredentialProvider.getCredentialEntry():66
org.apache.hadoop.conf.Configuration.getCredentialEntry():2397
org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders():2444
org.apache.hadoop.conf.Configuration.getPassword():2377
org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword():933
org.apache.hadoop.fs.s3a.S3AUtils.getPassword():918
org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword():884
org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword():842
org.apache.hadoop.fs.s3a.S3AUtils.getAWSAccessKeys():800
org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider.():66
jdk.internal.reflect.GeneratedConstructorAccessor79.newInstance():-1
jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance():45
java.lang.reflect.Constructor.newInstance():490
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider():706
org.apache.hadoop.fs.s3a.S3AUtils.buildAWSProviderList():659
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet():585
com.dremio.plugins.s3.store.S3FileSystem.createS3V1Client():223
com.dremio.plugins.s3.store.S3FileSystem$3.load():191
com.dremio.plugins.s3.store.S3FileSystem$3.load():187
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574
com.google.common.cache.LocalCache$Segment.loadSync():2316
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189
com.google.common.cache.LocalCache$Segment.get():2079
com.google.common.cache.LocalCache.get():4017
com.google.common.cache.LocalCache.getOrLoad():4040
com.google.common.cache.LocalCache$LocalLoadingCache.get():4989
com.dremio.plugins.s3.store.S3FileSystem.getS3V1Client():838
com.dremio.plugins.s3.store.S3FileSystem.getContainerCreators():289
com.dremio.plugins.util.ContainerFileSystem.refreshFileSystems():108
com.dremio.plugins.util.ContainerFileSystem.initialize():185
com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():296
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():423
org.apache.hadoop.security.UserGroupInformation.doAs():1899
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():301
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():272
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574
com.google.common.cache.LocalCache$Segment.loadSync():2316
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189
com.google.common.cache.LocalCache$Segment.get():2079
com.google.common.cache.LocalCache.get():4017
com.google.common.cache.LocalCache.getOrLoad():4040
com.google.common.cache.LocalCache$LocalLoadingCache.get():4989
com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():555
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():538
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():530
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():508
com.dremio.exec.store.dfs.FileSystemPlugin.start():965
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$4():787
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$6():865
java.util.concurrent.CompletableFuture$AsyncSupply.run():1700
java.util.concurrent.ThreadPoolExecutor.runWorker():1128
java.util.concurrent.ThreadPoolExecutor$Worker.run():628
java.lang.Thread.run():829
Caused By (io.netty.channel.ConnectTimeoutException) connection timed out after 30000 ms: ip-172-16-0-101.ec2.internal/172.16.0.101:34105
io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$2.run():615
io.netty.util.concurrent.PromiseTask.runTask():98
io.netty.util.concurrent.ScheduledFutureTask.run():156
io.netty.util.concurrent.AbstractEventExecutor.runTask():173
io.netty.util.concurrent.AbstractEventExecutor.safeExecute():166
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks():469
io.netty.channel.epoll.EpollEventLoop.run():408
io.netty.util.concurrent.SingleThreadEventExecutor$4.run():994
io.netty.util.internal.ThreadExecutorMap$2.run():74
io.netty.util.concurrent.FastThreadLocalRunnable.run():30
java.lang.Thread.run():829