# Unable to use OIDC based Auth with K8s Service Account while creating S3 source

**URL:** <https://community.dremio.com/t/unable-to-use-oidc-based-auth-with-k8s-service-account-while-creating-s3-source/10423>\
**Category:** Dremio University\
**Created:** [March 21, 2023, 11:06am UTC](https://community.dremio.com/t/unable-to-use-oidc-based-auth-with-k8s-service-account-while-creating-s3-source/10423 "2023-03-21T11:06:23Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![koushikgudipati](https://avatars.discourse-cdn.com/v4/letter/k/7bcc69/32.png) [@koushikgudipati](https://community.dremio.com/u/koushikgudipati)\
**Post date:** [March 21, 2023, 11:06am UTC](https://community.dremio.com/t/unable-to-use-oidc-based-auth-with-k8s-service-account-while-creating-s3-source/10423/1 "2023-03-21T11:06:23Z")

</div>

Hi Team

We are trying to connect to S3 source from dremio using kubernetes IAM role based auth for service accounts. We attached an IAM role to service account and dremio pod is running with that service account.  
When creating source we are selecting AWS Access key and we are only passing IAM role ARN.  
It throws error saying that it cannot able to connect.

 ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/f/f6037fefc0f63059b7d661273fe8004c4f94f59e.png)

We are not interested to use AWS keys and connect to S3 using IAM role and OIDC based auth.

Please let me know how to fix this issue.

Thanks

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [March 22, 2023, 6:45pm UTC](https://community.dremio.com/t/unable-to-use-oidc-based-auth-with-k8s-service-account-while-creating-s3-source/10423/2 "2023-03-22T18:45:33Z")

</div>

@koushikgudipati Does the server.log have a stack and says what the error is ?

---

<div class="post-metadata">

**Author:** ![koushikgudipati](https://avatars.discourse-cdn.com/v4/letter/k/7bcc69/32.png) [@koushikgudipati](https://community.dremio.com/u/koushikgudipati)\
**Post date:** [March 23, 2023, 9:19am UTC](https://community.dremio.com/t/unable-to-use-oidc-based-auth-with-k8s-service-account-while-creating-s3-source/10423/3 "2023-03-23T09:19:47Z")

</div>

@balaji.ramaswamy i see below error in logs

```auto
2023-03-23 08:51:42,559 [start-test-s3-source] WARN c.d.e.catalog.ManagedStoragePlugin - Error starting new source: test-s3-source
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Credential Verification failed.
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3953)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3976)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4960)
        at com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem(FileSystemPlugin.java:480)
        at com.dremio.exec.store.dfs.FileSystemPlugin.createFS(FileSystemPlugin.java:468)
        at com.dremio.exec.store.dfs.FileSystemPlugin.createFS(FileSystemPlugin.java:464)
        at com.dremio.exec.store.dfs.FileSystemPlugin.createFS(FileSystemPlugin.java:445)
        at com.dremio.exec.store.dfs.FileSystemPlugin.start(FileSystemPlugin.java:795)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$1(ManagedStoragePlugin.java:546)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$3(ManagedStoragePlugin.java:614)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Credential Verification failed.
        at com.dremio.plugins.s3.store.S3FileSystem.verifyCredentials(S3FileSystem.java:246)
        at com.dremio.plugins.s3.store.S3FileSystem.setup(S3FileSystem.java:226)
        at com.dremio.plugins.util.ContainerFileSystem.initialize(ContainerFileSystem.java:167)
        at com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0(FileSystemPlugin.java:240)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at com.dremio.exec.store.dfs.FileSystemPlugin$1.load(FileSystemPlugin.java:245)
        at com.dremio.exec.store.dfs.FileSystemPlugin$1.load(FileSystemPlugin.java:222)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
        ... 14 common frames omitted
Caused by: com.dremio.common.util.Retryer$OperationFailedAfterRetriesException: software.amazon.awssdk.services.sts.model.StsException: The security token included in the request is invalid. (Service: Sts, Status Code: 403, Request ID: ae30b14e-c985-47b7-bc02-c34797e29633, Extended Request ID: null)
        at com.dremio.common.util.Retryer.call(Retryer.java:60)
        at com.dremio.plugins.s3.store.S3FileSystem.verifyCredentials(S3FileSystem.java:240)
        ... 26 common frames omitted
Caused by: software.amazon.awssdk.services.sts.model.StsException: The security token included in the request is invalid. (Service: Sts, Status Code: 403, Request ID: ae30b14e-c985-47b7-bc02-c34797e29633, Extended Request ID: null)
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123)
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79)
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59)
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:135)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:161)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:114)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
        at software.amazon.awssdk.services.sts.DefaultStsClient.assumeRole(DefaultStsClient.java:260)
        at software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider.getUpdatedCredentials(StsAssumeRoleCredentialsProvider.java:67)
        at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.updateSessionCredentials(StsCredentialsProvider.java:80)
        at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:132)
        at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:89)
        at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.resolveCredentials(StsCredentialsProvider.java:91)
        at software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider.resolveCredentials(StsAssumeRoleCredentialsProvider.java:41)
        at com.dremio.plugins.s3.store.STSCredentialProviderV2.resolveCredentials(STSCredentialProviderV2.java:108)
        at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:79)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:99)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
        at software.amazon.awssdk.services.sts.DefaultStsClient.getCallerIdentity(DefaultStsClient.java:972)
        at com.dremio.plugins.s3.store.S3FileSystem.lambda$verifyCredentials$7(S3FileSystem.java:242)
        at com.dremio.common.util.Retryer.call(Retryer.java:56)
        ... 27 common frames omitted
2023-03-23 08:51:42,560 [start-test-s3-source] INFO c.d.s.s.LocalSchedulerService - Cancelling task metadata-refresh-wakeup-test-s3-source
2023-03-23 08:51:42,566 [qtp732929236-195] ERROR c.dremio.exec.catalog.PluginsManager - Exception while creating source.
com.dremio.common.exceptions.UserException: Could not connect to S3 source. Check your S3 data source settings and credentials.
        at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:885)
        at com.dremio.exec.catalog.ManagedStoragePlugin.createOrUpdateSource(ManagedStoragePlugin.java:411)
        at com.dremio.exec.catalog.ManagedStoragePlugin.createSource(ManagedStoragePlugin.java:318)
        at com.dremio.exec.catalog.PluginsManager.create(PluginsManager.java:172)
        at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:409)
        at com.dremio.exec.catalog.CatalogServiceImpl.access$400(CatalogServiceImpl.java:104)
        at com.dremio.exec.catalog.CatalogServiceImpl$SourceModifier.createSource(CatalogServiceImpl.java:742)
        at com.dremio.exec.catalog.CatalogImpl.createSource(CatalogImpl.java:1109)
        at com.dremio.exec.catalog.SourceAccessChecker.createSource(SourceAccessChecker.java:309)
        at com.dremio.exec.catalog.DelegatingCatalog.createSource(DelegatingCatalog.java:277)
        at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:162)
        at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:153)
        at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:149)
        at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:80)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
        at com.dremio.dac.server.GenericResponseHeadersFilter.doFilter(GenericResponseHeadersFilter.java:46)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at com.dremio.dac.server.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:66)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:713)
        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Credential Verification failed.
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
        at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin(ManagedStoragePlugin.java:1003)
        at com.dremio.exec.catalog.ManagedStoragePlugin.createOrUpdateSource(ManagedStoragePlugin.java:353)
        ... 73 common frames omitted
        Suppressed: java.lang.NullPointerException: null
                at com.dremio.exec.store.dfs.FileSystemPlugin.getSourceCapabilities(FileSystemPlugin.java:393)
                at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals(ManagedStoragePlugin.java:908)
                at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin(ManagedStoragePlugin.java:1018)
                ... 74 common frames omitted
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Credential Verification failed.
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3953)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3976)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4960)
        at com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem(FileSystemPlugin.java:480)
        at com.dremio.exec.store.dfs.FileSystemPlugin.createFS(FileSystemPlugin.java:468)
        at com.dremio.exec.store.dfs.FileSystemPlugin.createFS(FileSystemPlugin.java:464)
        at com.dremio.exec.store.dfs.FileSystemPlugin.createFS(FileSystemPlugin.java:445)
        at com.dremio.exec.store.dfs.FileSystemPlugin.start(FileSystemPlugin.java:795)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$1(ManagedStoragePlugin.java:546)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$3(ManagedStoragePlugin.java:614)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 common frames omitted
Caused by: java.lang.RuntimeException: Credential Verification failed.
        at com.dremio.plugins.s3.store.S3FileSystem.verifyCredentials(S3FileSystem.java:246)
        at com.dremio.plugins.s3.store.S3FileSystem.setup(S3FileSystem.java:226)
        at com.dremio.plugins.util.ContainerFileSystem.initialize(ContainerFileSystem.java:167)
        at com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0(FileSystemPlugin.java:240)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at com.dremio.exec.store.dfs.FileSystemPlugin$1.load(FileSystemPlugin.java:245)
        at com.dremio.exec.store.dfs.FileSystemPlugin$1.load(FileSystemPlugin.java:222)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
        ... 14 common frames omitted
Caused by: com.dremio.common.util.Retryer$OperationFailedAfterRetriesException: software.amazon.awssdk.services.sts.model.StsException: The security token included in the request is invalid. (Service: Sts, Status Code: 403, Request ID: ae30b14e-c985-47b7-bc02-c34797e29633, Extended Request ID: null)
        at com.dremio.common.util.Retryer.call(Retryer.java:60)
        at com.dremio.plugins.s3.store.S3FileSystem.verifyCredentials(S3FileSystem.java:240)
        ... 26 common frames omitted
Caused by: software.amazon.awssdk.services.sts.model.StsException: The security token included in the request is invalid. (Service: Sts, Status Code: 403, Request ID: ae30b14e-c985-47b7-bc02-c34797e29633, Extended Request ID: null)
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123)
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79)
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59)
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:135)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:161)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:114)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
        at software.amazon.awssdk.services.sts.DefaultStsClient.assumeRole(DefaultStsClient.java:260)
        at software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider.getUpdatedCredentials(StsAssumeRoleCredentialsProvider.java:67)
        at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.updateSessionCredentials(StsCredentialsProvider.java:80)
        at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:132)
        at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:89)
        at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.resolveCredentials(StsCredentialsProvider.java:91)
        at software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider.resolveCredentials(StsAssumeRoleCredentialsProvider.java:41)
        at com.dremio.plugins.s3.store.STSCredentialProviderV2.resolveCredentials(STSCredentialProviderV2.java:108)
        at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:79)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:99)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
        at software.amazon.awssdk.services.sts.DefaultStsClient.getCallerIdentity(DefaultStsClient.java:972)
        at com.dremio.plugins.s3.store.S3FileSystem.lambda$verifyCredentials$7(S3FileSystem.java:242)
        at com.dremio.common.util.Retryer.call(Retryer.java:56)
        ... 27 common frames omitted
2023-03-23 08:51:42,570 [qtp732929236-195] ERROR c.d.exec.catalog.CatalogServiceImpl - Exception encountered: Could not connect to S3 source. Check your S3 data source settings and credentials.
com.dremio.common.exceptions.UserException: Could not connect to S3 source. Check your S3 data source settings and credentials.
        at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:885)
        at com.dremio.exec.catalog.ManagedStoragePlugin.createOrUpdateSource(ManagedStoragePlugin.java:411)
        at com.dremio.exec.catalog.ManagedStoragePlugin.createSource(ManagedStoragePlugin.java:318)
        at com.dremio.exec.catalog.PluginsManager.create(PluginsManager.java:172)
        at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:409)
        at com.dremio.exec.catalog.CatalogServiceImpl.access$400(CatalogServiceImpl.java:104)
        at com.dremio.exec.catalog.CatalogServiceImpl$SourceModifier.createSource(CatalogServiceImpl.java:742)
        at com.dremio.exec.catalog.CatalogImpl.createSource(CatalogImpl.java:1109)
        at com.dremio.exec.catalog.SourceAccessChecker.createSource(SourceAccessChecker.java:309)
        at com.dremio.exec.catalog.DelegatingCatalog.createSource(DelegatingCatalog.java:277)
        at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:162)
        at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:153)
        at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:149)
        at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:80)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)

```

Can you confirm below  
While creating source when we choose “AWS Access Key” authentication, we see AWS Access keys and IAM role  
We can give either access keys or IAM role right?

i was giving dummy access keys and proper IAM role to test.

Thanks

---

<div class="post-metadata">

**Author:** ![koushikgudipati](https://avatars.discourse-cdn.com/v4/letter/k/7bcc69/32.png) [@koushikgudipati](https://community.dremio.com/u/koushikgudipati)\
**Post date:** [March 23, 2023, 10:15am UTC](https://community.dremio.com/t/unable-to-use-oidc-based-auth-with-k8s-service-account-while-creating-s3-source/10423/4 "2023-03-23T10:15:19Z")

</div>

Dremio version

Build  
24.0.0-202302100528110223-3a169b7c  
Edition  
Community Edition

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [March 29, 2023, 1:06am UTC](https://community.dremio.com/t/unable-to-use-oidc-based-auth-with-k8s-service-account-while-creating-s3-source/10423/5 "2023-03-29T01:06:35Z")

</div>

@koushikgudipati Can you follow steps in documentation?

You have to provide the right Access Key and IAM role

[https://docs.dremio.com/software/data-sources/s3/](https://docs.dremio.com/software/data-sources/s3/)

---

<div class="post-metadata">

**Author:** ![koushikgudipati](https://avatars.discourse-cdn.com/v4/letter/k/7bcc69/32.png) [@koushikgudipati](https://community.dremio.com/u/koushikgudipati)\
**Post date:** [March 29, 2023, 4:55am UTC](https://community.dremio.com/t/unable-to-use-oidc-based-auth-with-k8s-service-account-while-creating-s3-source/10423/6 "2023-03-29T04:55:45Z")

</div>

@balaji.ramaswamy

As per documentation it says “Access Key or IAM role”

```auto
#### Authentication

* AWS Access Key method – All or whitelisted (if specified) buckets associated with this access key or IAM role to assume (if specified) will be available. See [Advanced Options](https://docs.dremio.com/software/data-sources/s3/#advanced-options) for whitelisted information.
  * AWS Access Key – AWS access key.
  * AWS Access Secret – AWS access secret.
  * IAM Role to Assume – Used in conjunction with AWS Access Key method.

```

In my case i am trying to give only IAM role without AWS access key.  
AWS Access Key is mandatory for creating source?

Thanks  
Koushik

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [April 9, 2023, 12:11am UTC](https://community.dremio.com/t/unable-to-use-oidc-based-auth-with-k8s-service-account-while-creating-s3-source/10423/7 "2023-04-09T00:11:59Z")

</div>

@koushikgudipati There are 4 options

- AWS Key/Secret Key - In this scenario you need to give these 2 along with IAM role
- EC2 Metadata - Here you can only give IAM role but EC2 Metadata needs to be setup
- AWS Profile

[https://docs.dremio.com/software/data-sources/s3/#dremio-configuration](https://docs.dremio.com/software/data-sources/s3/#dremio-configuration)