Problem adding an s3 source

Jorge_Baez · December 1, 2017, 6:53pm

When trying to add an s3 source, after reading https://docs.dremio.com/data-sources/s3.html, i get the error “Failed to create workspaces for buckets owned by the account”

I don’t see what else is needed to connect apart from the access key and access secret, but i think that some kind of url is required in the field external buckets if i consider AWS Bucket Access

The full log of the error is:

017-12-01 18:44:29,350 [qtp915853613-140] ERROR c.d.d.server.GenericExceptionMapper - Unexpected exception when processing PUT http://MY_HOSTNAME:9047/apiv2/source/TEST_BUCKET_NAME?nocache=1512153871080 : java.lang.RuntimeException: Failed to create workspaces for buckets owned by the account.
java.lang.RuntimeException: Failed to create workspaces for buckets owned by the account.
at com.dremio.plugins.s3.store.S3FileSystem.initialize(S3FileSystem.java:140) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:181) ~[hadoop-common-2.8.0.jar:na]
at com.dremio.exec.store.dfs.FileSystemWrapper.(FileSystemWrapper.java:96) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.exec.util.ImpersonationUtil$2.run(ImpersonationUtil.java:206) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.exec.util.ImpersonationUtil$2.run(ImpersonationUtil.java:202) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_151]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_151]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807) ~[hadoop-common-2.8.0.jar:na]
at com.dremio.exec.util.ImpersonationUtil.createFileSystem(ImpersonationUtil.java:202) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.exec.util.ImpersonationUtil.createFileSystem(ImpersonationUtil.java:194) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.exec.store.dfs.FileSystemPlugin.getFS(FileSystemPlugin.java:249) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.exec.store.dfs.FileSystemStoragePlugin2.(FileSystemStoragePlugin2.java:92) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.plugins.s3.store.S3StoragePlugin2.(S3StoragePlugin2.java:34) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.plugins.s3.store.S3Plugin.getStoragePlugin2(S3Plugin.java:71) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.plugins.s3.store.S3Plugin.getStoragePlugin2(S3Plugin.java:45) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.exec.store.StoragePluginRegistryImpl.create(StoragePluginRegistryImpl.java:460) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.exec.store.StoragePluginRegistryImpl.createOrUpdate(StoragePluginRegistryImpl.java:288) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:124) ~[dremio-dac-backend-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:116) ~[dremio-dac-backend-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:81) ~[dremio-dac-backend-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 8D277BF211EF7B30)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221) ~[aws-java-sdk-s3-1.11.156.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168) ~[aws-java-sdk-s3-1.11.156.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4162) ~[aws-java-sdk-s3-1.11.156.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:914) ~[aws-java-sdk-s3-1.11.156.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:920) ~[aws-java-sdk-s3-1.11.156.jar:na]
at com.dremio.plugins.s3.store.S3FileSystem.initialize(S3FileSystem.java:110) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
… 69 common frames omitted

yufeldman · December 1, 2017, 7:01pm

Check if your AWS policy allows “listBuckets”. It seems that error comes from there:

balaji.ramaswamy · December 4, 2017, 4:45pm

Hi @Jorge_Baez

Wondering if adding the bucket policy as suggested by @yufeldman helped?

Thanks
@balaji.ramaswamy

Jorge_Baez · December 4, 2017, 5:07pm

Hi @balaji.ramaswamy,

We checked the policies and ListBucket is enabled, in fact, with that key/secret pair, we can upload/download and list all files in the bucket via awscli.

The error still persists.

balaji.ramaswamy · December 4, 2017, 5:14pm

Hi @Jorge_Baez

In your bucket policy (if not already added), please try adding the below and see if it works (replace samples.dremio.com with your public bucket name). Also please upload the error again if still does not work

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::samples.dremio.com"
        },
        {
            "Sid": "MakeItPublic",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::samples.dremio.com/*"
        }
    ]
}

Kindly let me know if this fixes the issue

Thanks,
@balaji.ramaswamy

Jorge_Baez · December 4, 2017, 5:48pm

From: http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-bucket-user-policy-specifying-principal-intro.html

Principal:
…
To grant permission to everyone, also referred as anonymous access, you set the wildcard, “*”, as the Principal value. For example, if you configure your bucket as a website, you want all the objects in the bucket to be publicly accessible.

I’m afraid this is not possible for our bucket.

Anyhow, i don’t fully understand why is it required to set the bucket as public if you provide the credentials with permissions to read/write/list the bucket.

The error is still the same after adding GetBucketLocation:

2017-12-04 17:39:26,437 [qtp915853613-673] ERROR c.d.d.server.GenericExceptionMapper - Unexpected exception when processing PUT http://MY_HOSTNAME:9047/apiv2/source/TEST_BUCKET_NAME?nocache=1512409168388 : java.lang.RuntimeException: Failed to create workspaces for buckets owned by the account.
java.lang.RuntimeException: Failed to create workspaces for buckets owned by the account.
at com.dremio.plugins.s3.store.S3FileSystem.initialize(S3FileSystem.java:140) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:181) ~[hadoop-common-2.8.0.jar:na]
at com.dremio.exec.store.dfs.FileSystemWrapper.(FileSystemWrapper.java:96) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.exec.util.ImpersonationUtil$2.run(ImpersonationUtil.java:206) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.exec.util.ImpersonationUtil$2.run(ImpersonationUtil.java:202) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_151]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_151]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807) ~[hadoop-common-2.8.0.jar:na]
at com.dremio.exec.util.ImpersonationUtil.createFileSystem(ImpersonationUtil.java:202) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.exec.util.ImpersonationUtil.createFileSystem(ImpersonationUtil.java:194) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.exec.store.dfs.FileSystemPlugin.getFS(FileSystemPlugin.java:249) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.exec.store.dfs.FileSystemStoragePlugin2.(FileSystemStoragePlugin2.java:92) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.plugins.s3.store.S3StoragePlugin2.(S3StoragePlugin2.java:34) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.plugins.s3.store.S3Plugin.getStoragePlugin2(S3Plugin.java:71) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.plugins.s3.store.S3Plugin.getStoragePlugin2(S3Plugin.java:45) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.exec.store.StoragePluginRegistryImpl.create(StoragePluginRegistryImpl.java:460) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.exec.store.StoragePluginRegistryImpl.createOrUpdate(StoragePluginRegistryImpl.java:288) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:124) ~[dremio-dac-backend-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:116) ~[dremio-dac-backend-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:81) ~[dremio-dac-backend-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 6E73AAC8CA5049FB)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221) ~[aws-java-sdk-s3-1.11.156.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168) ~[aws-java-sdk-s3-1.11.156.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4162) ~[aws-java-sdk-s3-1.11.156.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:914) ~[aws-java-sdk-s3-1.11.156.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:920) ~[aws-java-sdk-s3-1.11.156.jar:na]
at com.dremio.plugins.s3.store.S3FileSystem.initialize(S3FileSystem.java:110) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
… 69 common frames omitted

balaji.ramaswamy · December 4, 2017, 7:07pm

Hi @Jorge_Baez

Sorry did not realize it was not a public bucket. Let me get back to you on this shortly

Thanks
@balaji.ramaswamy

balaji.ramaswamy · December 5, 2017, 6:14pm

HI @Jorge_Baez

One other thing, have you defined any list bucket policies under the IAM policy of the user trying to access this bucket?

Thanks,
@balaji.ramaswamy

Jorge_Baez · December 5, 2017, 6:30pm

Hi @balaji.ramaswamy

Yes, list/get/delete/put actions are enabled for the user, since this key/secret pair is currently being used from awscli and some custom python scripts, using boto. In both cases directories are listed, and files are uploaded/downloaded with the same key/secret pair configured in Dremio.

Just to be sure:
The field “Name” is just a descriptive name for the source, and apart from the fields of the key and the secret, in the external buckets section, the name of the bucket should be provided in some special form? We tried with different variations of s3://bucketname and also just the bucketname, and none of them worked.

Topic		Replies	Views
Error adding S3 source	2	2092	September 12, 2018
Reading data from S3 failing after upgrade to 1.2.2	8	2253	December 14, 2017
Issue in Connecting S3 as DataSource	4	1439	February 25, 2020
AWS Bucket Access	7	3054	July 21, 2017
Adding S3 source with IAM role	2	1743	August 5, 2020

Problem adding an s3 source

Related topics