# The sourceis currently unavailable. Metadata is not accessible **URL:** \ **Category:** Uncategorized\ **Created:** [October 26, 2024, 12:13am UTC](https://community.dremio.com/t/the-sourceis-currently-unavailable-metadata-is-not-accessible/12439 "2024-10-26T00:13:02Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![pablonqn](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/pablonqn/32/6099_2.png) [@pablonqn](https://community.dremio.com/u/pablonqn)\ **Post date:** [October 26, 2024, 12:13am UTC](https://community.dremio.com/t/the-sourceis-currently-unavailable-metadata-is-not-accessible/12439/1 "2024-10-26T00:13:02Z")
Hi all I have this exception when we run the query to our S3 bucket, where we have each tenant’s parquets. We can create the data source and connect to S3 without a problem; however, once the query runs, we get the exception below. We have 1 coordinator node and 2 workers If some one could guide me to fix this issue , I will really glad Thanks Pablo VALIDATION ERROR: Failure getting source [tenant\_0007]. ErrorOrigin: EXECUTOR [Error Id: 9ea8842b-1641-43c6-bd8f-e13777034bf4] (java.util.concurrent.ExecutionException) java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider instantiation exception: java.lang.RuntimeException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception java.util.concurrent.CompletableFuture.reportGet():395 java.util.concurrent.CompletableFuture.get():2022 com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin():1563 com.dremio.exec.catalog.ManagedStoragePlugin.replacePluginWithLock():1498 com.dremio.exec.catalog.PluginsManager.getSynchronized():557 com.dremio.exec.catalog.CatalogServiceImpl.getPlugin():670 com.dremio.exec.catalog.CatalogServiceImpl.getSource():731 com.dremio.sabot.exec.fragment.FragmentExecutionContext.getStoragePlugin():77 com.dremio.exec.store.parquet.ParquetSplitReaderCreatorIterator.():194 com.dremio.exec.store.parquet.ParquetSplitReaderCreatorIterator.():156 com.dremio.exec.store.parquet.ParquetOperatorCreator.create():42 com.dremio.exec.store.parquet.ParquetOperatorCreator.create():29 com.dremio.sabot.driver.OperatorCreatorRegistry.getProducerOperator():132 com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan():241 com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan():123 com.dremio.exec.physical.base.AbstractSubScan.accept():81 com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():308 com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():123 com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject():84 com.dremio.exec.physical.config.Project.accept():54 com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():308 com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():123 com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject():84 com.dremio.exec.physical.config.Project.accept():54 com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():308 com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():123 com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriter():69 com.dremio.exec.physical.base.AbstractWriter.accept():30 com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSender():205 com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSender():123 com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitSingleSender():201 com.dremio.exec.physical.config.SingleSender.accept():77 com.dremio.sabot.driver.PipelineCreator.get():109 com.dremio.sabot.driver.PipelineCreator.get():103 com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():724 com.dremio.sabot.exec.fragment.FragmentExecutor.run():514 com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():1234 com.dremio.sabot.task.AsyncTaskWrapper.run():130 com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():279 com.dremio.sabot.task.slicing.SlicingThread.run():186 Caused By (java.io.IOException) java.util.concurrent.ExecutionException: java.io.IOException: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider instantiation exception: java.lang.RuntimeException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception com.dremio.plugins.s3.store.S3FileSystem.getContainerCreators():311 com.dremio.plugins.util.ContainerFileSystem.refreshFileSystems():108 com.dremio.plugins.util.ContainerFileSystem.initialize():185 com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():296 java.security.AccessController.doPrivileged():-2 javax.security.auth.Subject.doAs():423 org.apache.hadoop.security.UserGroupInformation.doAs():1899 com.dremio.exec.store.dfs.FileSystemPlugin$1.load():301 com.dremio.exec.store.dfs.FileSystemPlugin$1.load():272 com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574 com.google.common.cache.LocalCache$Segment.loadSync():2316 com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189 com.google.common.cache.LocalCache$Segment.get():2079 com.google.common.cache.LocalCache.get():4017 com.google.common.cache.LocalCache.getOrLoad():4040 com.google.common.cache.LocalCache$LocalLoadingCache.get():4989 com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():555 com.dremio.exec.store.dfs.FileSystemPlugin.createFS():538 com.dremio.exec.store.dfs.FileSystemPlugin.createFS():530 com.dremio.exec.store.dfs.FileSystemPlugin.createFS():508 com.dremio.exec.store.dfs.FileSystemPlugin.start():965 com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$4():787 com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$6():865 java.util.concurrent.CompletableFuture$AsyncSupply.run():1700 java.util.concurrent.ThreadPoolExecutor.runWorker():1128 java.util.concurrent.ThreadPoolExecutor$Worker.run():628 java.lang.Thread.run():829 Caused By (java.util.concurrent.ExecutionException) java.io.IOException: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider instantiation exception: java.lang.RuntimeException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception com.google.common.util.concurrent.AbstractFuture.getDoneValue():594 com.google.common.util.concurrent.AbstractFuture.get():553 com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get():110 com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly():247 com.google.common.cache.LocalCache$Segment.getAndRecordStats():2349 com.google.common.cache.LocalCache$Segment.loadSync():2317 com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189 com.google.common.cache.LocalCache$Segment.get():2079 com.google.common.cache.LocalCache.get():4017 com.google.common.cache.LocalCache.getOrLoad():4040 com.google.common.cache.LocalCache$LocalLoadingCache.get():4989 com.dremio.plugins.s3.store.S3FileSystem.getS3V1Client():838 com.dremio.plugins.s3.store.S3FileSystem.getContainerCreators():289 com.dremio.plugins.util.ContainerFileSystem.refreshFileSystems():108 com.dremio.plugins.util.ContainerFileSystem.initialize():185 com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():296 java.security.AccessController.doPrivileged():-2 javax.security.auth.Subject.doAs():423 org.apache.hadoop.security.UserGroupInformation.doAs():1899 com.dremio.exec.store.dfs.FileSystemPlugin$1.load():301 com.dremio.exec.store.dfs.FileSystemPlugin$1.load():272 com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574 com.google.common.cache.LocalCache$Segment.loadSync():2316 com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189 com.google.common.cache.LocalCache$Segment.get():2079 com.google.common.cache.LocalCache.get():4017 com.google.common.cache.LocalCache.getOrLoad():4040 com.google.common.cache.LocalCache$LocalLoadingCache.get():4989 com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():555 com.dremio.exec.store.dfs.FileSystemPlugin.createFS():538 com.dremio.exec.store.dfs.FileSystemPlugin.createFS():530 com.dremio.exec.store.dfs.FileSystemPlugin.createFS():508 com.dremio.exec.store.dfs.FileSystemPlugin.start():965 com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$4():787 com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$6():865 java.util.concurrent.CompletableFuture$AsyncSupply.run():1700 java.util.concurrent.ThreadPoolExecutor.runWorker():1128 java.util.concurrent.ThreadPoolExecutor$Worker.run():628 java.lang.Thread.run():829 Caused By (java.io.IOException) org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider instantiation exception: java.lang.RuntimeException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider():749 org.apache.hadoop.fs.s3a.S3AUtils.buildAWSProviderList():659 org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet():585 com.dremio.plugins.s3.store.S3FileSystem.createS3V1Client():223 com.dremio.plugins.s3.store.S3FileSystem$3.load():191 com.dremio.plugins.s3.store.S3FileSystem$3.load():187 com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574 com.google.common.cache.LocalCache$Segment.loadSync():2316 com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189 com.google.common.cache.LocalCache$Segment.get():2079 com.google.common.cache.LocalCache.get():4017 com.google.common.cache.LocalCache.getOrLoad():4040 com.google.common.cache.LocalCache$LocalLoadingCache.get():4989 com.dremio.plugins.s3.store.S3FileSystem.getS3V1Client():838 com.dremio.plugins.s3.store.S3FileSystem.getContainerCreators():289 com.dremio.plugins.util.ContainerFileSystem.refreshFileSystems():108 com.dremio.plugins.util.ContainerFileSystem.initialize():185 com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():296 java.security.AccessController.doPrivileged():-2 javax.security.auth.Subject.doAs():423 org.apache.hadoop.security.UserGroupInformation.doAs():1899 com.dremio.exec.store.dfs.FileSystemPlugin$1.load():301 com.dremio.exec.store.dfs.FileSystemPlugin$1.load():272 com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574 com.google.common.cache.LocalCache$Segment.loadSync():2316 com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189 com.google.common.cache.LocalCache$Segment.get():2079 com.google.common.cache.LocalCache.get():4017 com.google.common.cache.LocalCache.getOrLoad():4040 com.google.common.cache.LocalCache$LocalLoadingCache.get():4989 com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():555 com.dremio.exec.store.dfs.FileSystemPlugin.createFS():538 com.dremio.exec.store.dfs.FileSystemPlugin.createFS():530 com.dremio.exec.store.dfs.FileSystemPlugin.createFS():508 com.dremio.exec.store.dfs.FileSystemPlugin.start():965 com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$4():787 com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$6():865 java.util.concurrent.CompletableFuture$AsyncSupply.run():1700 java.util.concurrent.ThreadPoolExecutor.runWorker():1128 java.util.concurrent.ThreadPoolExecutor$Worker.run():628 java.lang.Thread.run():829 Caused By (java.lang.RuntimeException) io.grpc.StatusRuntimeException: UNAVAILABLE: io exception com.dremio.services.credentials.CredentialsProviderContextImpl.lookupOnLeader():68 com.dremio.services.credentials.SystemSecretCredentialsProvider.lambda$doLookup$0():78 com.dremio.telemetry.api.metrics.TimerUtils.timedExceptionThrowingOperation():124 com.dremio.services.credentials.SystemSecretCredentialsProvider.doLookup():75 com.dremio.services.credentials.AbstractSimpleCredentialsProvider.lookup():41 com.dremio.services.credentials.CredentialsServiceImpl.lookup():95 com.dremio.hadoop.security.alias.DremioCredentialProvider.lambda$resolveCredentialPattern$0():109 com.dremio.telemetry.api.metrics.TimerUtils.timedExceptionThrowingOperation():124 com.dremio.hadoop.security.alias.DremioCredentialProvider.resolveCredentialPattern():103 com.dremio.hadoop.security.alias.DremioCredentialProvider.getCredentialEntry():66 org.apache.hadoop.conf.Configuration.getCredentialEntry():2397 org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders():2444 org.apache.hadoop.conf.Configuration.getPassword():2377 org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword():933 org.apache.hadoop.fs.s3a.S3AUtils.getPassword():918 org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword():884 org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword():842 org.apache.hadoop.fs.s3a.S3AUtils.getAWSAccessKeys():800 org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider.():66 jdk.internal.reflect.GeneratedConstructorAccessor79.newInstance():-1 jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance():45 java.lang.reflect.Constructor.newInstance():490 org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider():706 org.apache.hadoop.fs.s3a.S3AUtils.buildAWSProviderList():659 org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet():585 com.dremio.plugins.s3.store.S3FileSystem.createS3V1Client():223 com.dremio.plugins.s3.store.S3FileSystem$3.load():191 com.dremio.plugins.s3.store.S3FileSystem$3.load():187 com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574 com.google.common.cache.LocalCache$Segment.loadSync():2316 com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189 com.google.common.cache.LocalCache$Segment.get():2079 com.google.common.cache.LocalCache.get():4017 com.google.common.cache.LocalCache.getOrLoad():4040 com.google.common.cache.LocalCache$LocalLoadingCache.get():4989 com.dremio.plugins.s3.store.S3FileSystem.getS3V1Client():838 com.dremio.plugins.s3.store.S3FileSystem.getContainerCreators():289 com.dremio.plugins.util.ContainerFileSystem.refreshFileSystems():108 com.dremio.plugins.util.ContainerFileSystem.initialize():185 com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():296 java.security.AccessController.doPrivileged():-2 javax.security.auth.Subject.doAs():423 org.apache.hadoop.security.UserGroupInformation.doAs():1899 com.dremio.exec.store.dfs.FileSystemPlugin$1.load():301 com.dremio.exec.store.dfs.FileSystemPlugin$1.load():272 com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574 com.google.common.cache.LocalCache$Segment.loadSync():2316 com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189 com.google.common.cache.LocalCache$Segment.get():2079 com.google.common.cache.LocalCache.get():4017 com.google.common.cache.LocalCache.getOrLoad():4040 com.google.common.cache.LocalCache$LocalLoadingCache.get():4989 com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():555 com.dremio.exec.store.dfs.FileSystemPlugin.createFS():538 com.dremio.exec.store.dfs.FileSystemPlugin.createFS():530 com.dremio.exec.store.dfs.FileSystemPlugin.createFS():508 com.dremio.exec.store.dfs.FileSystemPlugin.start():965 com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$4():787 com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$6():865 java.util.concurrent.CompletableFuture$AsyncSupply.run():1700 java.util.concurrent.ThreadPoolExecutor.runWorker():1128 java.util.concurrent.ThreadPoolExecutor$Worker.run():628 java.lang.Thread.run():829 Caused By (io.grpc.StatusRuntimeException) UNAVAILABLE: io exception io.grpc.stub.ClientCalls.toStatusRuntimeException():268 io.grpc.stub.ClientCalls.getUnchecked():249 io.grpc.stub.ClientCalls.blockingUnaryCall():167 com.dremio.services.credentials.proto.CredentialsServiceGrpc$CredentialsServiceBlockingStub.lookup():160 com.dremio.services.credentials.CredentialsProviderContextImpl.lookupOnLeader():58 com.dremio.services.credentials.SystemSecretCredentialsProvider.lambda$doLookup$0():78 com.dremio.telemetry.api.metrics.TimerUtils.timedExceptionThrowingOperation():124 com.dremio.services.credentials.SystemSecretCredentialsProvider.doLookup():75 com.dremio.services.credentials.AbstractSimpleCredentialsProvider.lookup():41 com.dremio.services.credentials.CredentialsServiceImpl.lookup():95 com.dremio.hadoop.security.alias.DremioCredentialProvider.lambda$resolveCredentialPattern$0():109 com.dremio.telemetry.api.metrics.TimerUtils.timedExceptionThrowingOperation():124 com.dremio.hadoop.security.alias.DremioCredentialProvider.resolveCredentialPattern():103 com.dremio.hadoop.security.alias.DremioCredentialProvider.getCredentialEntry():66 org.apache.hadoop.conf.Configuration.getCredentialEntry():2397 org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders():2444 org.apache.hadoop.conf.Configuration.getPassword():2377 org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword():933 org.apache.hadoop.fs.s3a.S3AUtils.getPassword():918 org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword():884 org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword():842 org.apache.hadoop.fs.s3a.S3AUtils.getAWSAccessKeys():800 org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider.():66 jdk.internal.reflect.GeneratedConstructorAccessor79.newInstance():-1 jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance():45 java.lang.reflect.Constructor.newInstance():490 org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider():706 org.apache.hadoop.fs.s3a.S3AUtils.buildAWSProviderList():659 org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet():585 com.dremio.plugins.s3.store.S3FileSystem.createS3V1Client():223 com.dremio.plugins.s3.store.S3FileSystem$3.load():191 com.dremio.plugins.s3.store.S3FileSystem$3.load():187 com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574 com.google.common.cache.LocalCache$Segment.loadSync():2316 com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189 com.google.common.cache.LocalCache$Segment.get():2079 com.google.common.cache.LocalCache.get():4017 com.google.common.cache.LocalCache.getOrLoad():4040 com.google.common.cache.LocalCache$LocalLoadingCache.get():4989 com.dremio.plugins.s3.store.S3FileSystem.getS3V1Client():838 com.dremio.plugins.s3.store.S3FileSystem.getContainerCreators():289 com.dremio.plugins.util.ContainerFileSystem.refreshFileSystems():108 com.dremio.plugins.util.ContainerFileSystem.initialize():185 com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():296 java.security.AccessController.doPrivileged():-2 javax.security.auth.Subject.doAs():423 org.apache.hadoop.security.UserGroupInformation.doAs():1899 com.dremio.exec.store.dfs.FileSystemPlugin$1.load():301 com.dremio.exec.store.dfs.FileSystemPlugin$1.load():272 com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3574 com.google.common.cache.LocalCache$Segment.loadSync():2316 com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2189 com.google.common.cache.LocalCache$Segment.get():2079 com.google.common.cache.LocalCache.get():4017 com.google.common.cache.LocalCache.getOrLoad():4040 com.google.common.cache.LocalCache$LocalLoadingCache.get():4989 com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():555 com.dremio.exec.store.dfs.FileSystemPlugin.createFS():538 com.dremio.exec.store.dfs.FileSystemPlugin.createFS():530 com.dremio.exec.store.dfs.FileSystemPlugin.createFS():508 com.dremio.exec.store.dfs.FileSystemPlugin.start():965 com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$4():787 com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$6():865 java.util.concurrent.CompletableFuture$AsyncSupply.run():1700 java.util.concurrent.ThreadPoolExecutor.runWorker():1128 java.util.concurrent.ThreadPoolExecutor$Worker.run():628 java.lang.Thread.run():829 Caused By (io.netty.channel.ConnectTimeoutException) connection timed out after 30000 ms: ip-172-16-0-101.ec2.internal/172.16.0.101:34105 io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$2.run():615 io.netty.util.concurrent.PromiseTask.runTask():98 io.netty.util.concurrent.ScheduledFutureTask.run():156 io.netty.util.concurrent.AbstractEventExecutor.runTask():173 io.netty.util.concurrent.AbstractEventExecutor.safeExecute():166 io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks():469 io.netty.channel.epoll.EpollEventLoop.run():408 io.netty.util.concurrent.SingleThreadEventExecutor$4.run():994 io.netty.util.internal.ThreadExecutorMap$2.run():74 io.netty.util.concurrent.FastThreadLocalRunnable.run():30 java.lang.Thread.run():829 ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [October 27, 2024, 11:07pm UTC](https://community.dremio.com/t/the-sourceis-currently-unavailable-metadata-is-not-accessible/12439/2 "2024-10-27T23:07:29Z")
@pablonqn It looks liek the executor is unable to query the source, can you please restart the pod/node and on startup tail the server.log to see if the source shows as Healthy, also please send the server.log once Dremio is restarted and comes up (from both workers)