When trying to add a china s3 source, I get the error “The security token included in the request is invalid”.
The full log if the error is:
2019-09-20 02:48:33,416 [qtp1092465037-88] INFO c.d.exec.catalog.CatalogServiceImpl - User Error Occurred [ErrorId: 51bb48b4-514e-4c26-a7f5-5bcf69e85780]
com.dremio.common.exceptions.UserException: Failure while configuring source [S3].
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:776) ~[dremio-common-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.exec.catalog.CatalogServiceImpl.createOrUpdateSource(CatalogServiceImpl.java:671) [dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:406) [dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.exec.catalog.CatalogServiceImpl.access$600(CatalogServiceImpl.java:105) [dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.exec.catalog.CatalogServiceImpl$SourceModifier.createSource(CatalogServiceImpl.java:1023) [dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.exec.catalog.CatalogImpl.createSource(CatalogImpl.java:608) [dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.exec.catalog.SourceAccessChecker.createSource(SourceAccessChecker.java:245) [dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.exec.catalog.DelegatingCatalog.createSource(DelegatingCatalog.java:206) [dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:157) [dremio-dac-backend-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:148) [dremio-dac-backend-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:144) [dremio-dac-backend-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:80) [dremio-dac-backend-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) [jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) [jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at com.dremio.dac.server.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:52) [dremio-dac-backend-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) [jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:51) [jetty-servlets-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) [jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) [jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) [jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) [jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) [jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56) [jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.Server.handle(Server.java:502) [jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) [jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) [jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) [jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) [jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) [jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) [jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
Caused by: java.util.concurrent.ExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Credential Verification failed. Exception: The security token included in the request is invalid. (Service: Sts, Status Code: 403, Request ID: 22ba01c3-db51-11e9-99dd-61d75ec76a21)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500) ~[guava-20.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:401) ~[guava-20.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) ~[guava-20.0.jar:na]
at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:68) ~[guava-20.0.jar:na]
at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:104) ~[guava-20.0.jar:na]
at com.dremio.exec.catalog.CatalogServiceImpl.createOrUpdateSource(CatalogServiceImpl.java:659) [dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
… 65 common frames omitted
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Credential Verification failed. Exception: The security token included in the request is invalid. (Service: Sts, Status Code: 403, Request ID: 22ba01c3-db51-11e9-99dd-61d75ec76a21)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214) ~[guava-20.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:4053) ~[guava-20.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057) ~[guava-20.0.jar:na]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986) ~[guava-20.0.jar:na]
at com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem(FileSystemPlugin.java:289) ~[dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.exec.store.dfs.FileSystemPlugin.createFS(FileSystemPlugin.java:277) ~[dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.exec.store.dfs.FileSystemPlugin.createFS(FileSystemPlugin.java:273) ~[dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.exec.store.dfs.FileSystemPlugin.createFS(FileSystemPlugin.java:269) ~[dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.exec.store.dfs.FileSystemPlugin.start(FileSystemPlugin.java:562) ~[dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.exec.catalog.ManagedStoragePlugin$1.run(ManagedStoragePlugin.java:257) ~[dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.concurrent.RenamingRunnable.run(RenamingRunnable.java:36) ~[dremio-common-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.concurrent.SingletonRunnable.run(SingletonRunnable.java:41) ~[dremio-common-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.concurrent.SafeRunnable.run(SafeRunnable.java:40) ~[dremio-common-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.concurrent.Runnables$1.run(Runnables.java:45) ~[dremio-common-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
Caused by: java.lang.RuntimeException: Credential Verification failed. Exception: The security token included in the request is invalid. (Service: Sts, Status Code: 403, Request ID: 22ba01c3-db51-11e9-99dd-61d75ec76a21)
at com.dremio.plugins.s3.store.S3FileSystem.verifyCredentials(S3FileSystem.java:201) ~[dremio-s3-plugin-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.plugins.s3.store.S3FileSystem.setup(S3FileSystem.java:174) ~[dremio-s3-plugin-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.plugins.util.ContainerFileSystem.initialize(ContainerFileSystem.java:156) ~[dremio-s3-plugin-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0(FileSystemPlugin.java:195) ~[dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_60]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_60]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) ~[hadoop-common-3.2.0-dremio-201909041046110436-7295d98.jar:na]
at com.dremio.exec.store.dfs.FileSystemPlugin$1.load(FileSystemPlugin.java:200) ~[dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.dremio.exec.store.dfs.FileSystemPlugin$1.load(FileSystemPlugin.java:177) ~[dremio-sabot-kernel-4.0.0-201909121834570395-c7a5071.jar:4.0.0-201909121834570395-c7a5071]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628) ~[guava-20.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336) ~[guava-20.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295) ~[guava-20.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208) ~[guava-20.0.jar:na]
… 13 common frames omitted
I tested it with the aws client, when I configured the default region name with “cn-north-1”, the connection was successful.
when I configured the other default region name, the connection failed.
It may be similar to China AWS S3 Connection.
Can I configure default region of s3 in dremio 4.0? Or what can I do to solve this problem?
Thanks.