# Problem adding an s3 source

**URL:** <https://community.dremio.com/t/problem-adding-an-s3-source/514>\
**Category:** Uncategorized\
**Created:** [December 1, 2017, 6:53pm UTC](https://community.dremio.com/t/problem-adding-an-s3-source/514 "2017-12-01T18:53:48Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jorge\_Baez](https://avatars.discourse-cdn.com/v4/letter/j/d78d45/32.png) [@Jorge\_Baez](https://community.dremio.com/u/Jorge_Baez)\
**Post date:** [December 1, 2017, 6:53pm UTC](https://community.dremio.com/t/problem-adding-an-s3-source/514/1 "2017-12-01T18:53:48Z")

</div>

When trying to add an s3 source, after reading [https://docs.dremio.com/data-sources/s3.html](https://docs.dremio.com/data-sources/s3.html), i get the error “Failed to create workspaces for buckets owned by the account”

I don’t see what else is needed to connect apart from the access key and access secret, but i think that some kind of url is required in the field external buckets if i consider [AWS Bucket Access](https://community.dremio.com/t/aws-bucket-access/67/5)

The full log of the error is:

017-12-01 18:44:29,350 [qtp915853613-140] ERROR c.d.d.server.GenericExceptionMapper - Unexpected exception when processing PUT http://MY\_HOSTNAME:9047/apiv2/source/TEST\_BUCKET\_NAME?nocache=1512153871080 : java.lang.RuntimeException: Failed to create workspaces for buckets owned by the account.  
java.lang.RuntimeException: Failed to create workspaces for buckets owned by the account.  
at com.dremio.plugins.s3.store.S3FileSystem.initialize(S3FileSystem.java:140) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811) ~[hadoop-common-2.8.0.jar:na]  
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) ~[hadoop-common-2.8.0.jar:na]  
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848) ~[hadoop-common-2.8.0.jar:na]  
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830) ~[hadoop-common-2.8.0.jar:na]  
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) ~[hadoop-common-2.8.0.jar:na]  
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:181) ~[hadoop-common-2.8.0.jar:na]  
at com.dremio.exec.store.dfs.FileSystemWrapper.(FileSystemWrapper.java:96) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.exec.util.ImpersonationUtil$2.run(ImpersonationUtil.java:206) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.exec.util.ImpersonationUtil$2.run(ImpersonationUtil.java:202) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0\_151]  
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0\_151]  
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807) ~[hadoop-common-2.8.0.jar:na]  
at com.dremio.exec.util.ImpersonationUtil.createFileSystem(ImpersonationUtil.java:202) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.exec.util.ImpersonationUtil.createFileSystem(ImpersonationUtil.java:194) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.exec.store.dfs.FileSystemPlugin.getFS(FileSystemPlugin.java:249) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.exec.store.dfs.FileSystemStoragePlugin2.(FileSystemStoragePlugin2.java:92) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.plugins.s3.store.S3StoragePlugin2.(S3StoragePlugin2.java:34) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.plugins.s3.store.S3Plugin.getStoragePlugin2(S3Plugin.java:71) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.plugins.s3.store.S3Plugin.getStoragePlugin2(S3Plugin.java:45) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.exec.store.StoragePluginRegistryImpl.create(StoragePluginRegistryImpl.java:460) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.exec.store.StoragePluginRegistryImpl.createOrUpdate(StoragePluginRegistryImpl.java:288) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:124) ~[dremio-dac-backend-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:116) ~[dremio-dac-backend-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:81) ~[dremio-dac-backend-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source) ~[na:na]  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0\_151]  
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0\_151]  
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]  
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]  
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]  
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]  
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]  
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]  
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]  
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]  
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]  
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]  
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]  
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]  
at java.lang.Thread.run(Thread.java:748) [na:1.8.0\_151]  
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 8D277BF211EF7B30)  
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221) ~[aws-java-sdk-s3-1.11.156.jar:na]  
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168) ~[aws-java-sdk-s3-1.11.156.jar:na]  
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4162) ~[aws-java-sdk-s3-1.11.156.jar:na]  
at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:914) ~[aws-java-sdk-s3-1.11.156.jar:na]  
at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:920) ~[aws-java-sdk-s3-1.11.156.jar:na]  
at com.dremio.plugins.s3.store.S3FileSystem.initialize(S3FileSystem.java:110) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
… 69 common frames omitted

---

<div class="post-metadata">

**Author:** ![yufeldman](https://avatars.discourse-cdn.com/v4/letter/y/c77e96/32.png) [@yufeldman](https://community.dremio.com/u/yufeldman)\
**Post date:** [December 1, 2017, 7:01pm UTC](https://community.dremio.com/t/problem-adding-an-s3-source/514/2 "2017-12-01T19:01:40Z")

</div>

Check if your AWS policy allows “listBuckets”. It seems that error comes from there:

> [@Jorge\_Baez](#):
>
> Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 8D277BF211EF7B30)

> [@Jorge\_Baez](#):
>
> at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:914) ~[aws-java-sdk-s3-1.11.156.jar:na]
> 
> at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:920) ~[aws-java-sdk-s3-1.11.156.jar:na]

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [December 4, 2017, 4:45pm UTC](https://community.dremio.com/t/problem-adding-an-s3-source/514/3 "2017-12-04T16:45:27Z")

</div>

Hi @Jorge_Baez

Wondering if adding the bucket policy as suggested by @yufeldman helped?

Thanks  
@balaji.ramaswamy

---

<div class="post-metadata">

**Author:** ![Jorge\_Baez](https://avatars.discourse-cdn.com/v4/letter/j/d78d45/32.png) [@Jorge\_Baez](https://community.dremio.com/u/Jorge_Baez)\
**Post date:** [December 4, 2017, 5:07pm UTC](https://community.dremio.com/t/problem-adding-an-s3-source/514/4 "2017-12-04T17:07:03Z")

</div>

Hi @balaji.ramaswamy,

We checked the policies and ListBucket is enabled, in fact, with that key/secret pair, we can upload/download and list all files in the bucket via awscli.

The error still persists.

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [December 4, 2017, 5:14pm UTC](https://community.dremio.com/t/problem-adding-an-s3-source/514/5 "2017-12-04T17:14:56Z")

</div>

Hi @Jorge_Baez

In your bucket policy (if not already added), please try adding the below and see if it works (replace [samples.dremio.com](http://samples.dremio.com) with your public bucket name). Also please upload the error again if still does not work

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::samples.dremio.com"
        },
        {
            "Sid": "MakeItPublic",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::samples.dremio.com/*"
        }
    ]
} 

```

Kindly let me know if this fixes the issue

Thanks,  
@balaji.ramaswamy

---

<div class="post-metadata">

**Author:** ![Jorge\_Baez](https://avatars.discourse-cdn.com/v4/letter/j/d78d45/32.png) [@Jorge\_Baez](https://community.dremio.com/u/Jorge_Baez)\
**Post date:** [December 4, 2017, 5:48pm UTC](https://community.dremio.com/t/problem-adding-an-s3-source/514/6 "2017-12-04T17:48:48Z")

</div>

From: [http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-bucket-user-policy-specifying-principal-intro.html](http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-bucket-user-policy-specifying-principal-intro.html)

Principal:  
…  
To grant permission to everyone, also referred as anonymous access, you set the wildcard, “\*”, as the Principal value. For example, if you configure your bucket as a website, you want all the objects in the bucket to be publicly accessible.

I’m afraid this is not possible for our bucket.

Anyhow, i don’t fully understand why is it required to set the bucket as public if you provide the credentials with permissions to read/write/list the bucket.

The error is still the same after adding GetBucketLocation:

2017-12-04 17:39:26,437 [qtp915853613-673] ERROR c.d.d.server.GenericExceptionMapper - Unexpected exception when processing PUT http://MY\_HOSTNAME:9047/apiv2/source/TEST\_BUCKET\_NAME?nocache=1512409168388 : java.lang.RuntimeException: Failed to create workspaces for buckets owned by the account.  
java.lang.RuntimeException: Failed to create workspaces for buckets owned by the account.  
at com.dremio.plugins.s3.store.S3FileSystem.initialize(S3FileSystem.java:140) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811) ~[hadoop-common-2.8.0.jar:na]  
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) ~[hadoop-common-2.8.0.jar:na]  
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848) ~[hadoop-common-2.8.0.jar:na]  
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830) ~[hadoop-common-2.8.0.jar:na]  
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) ~[hadoop-common-2.8.0.jar:na]  
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:181) ~[hadoop-common-2.8.0.jar:na]  
at com.dremio.exec.store.dfs.FileSystemWrapper.(FileSystemWrapper.java:96) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.exec.util.ImpersonationUtil$2.run(ImpersonationUtil.java:206) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.exec.util.ImpersonationUtil$2.run(ImpersonationUtil.java:202) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0\_151]  
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0\_151]  
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807) ~[hadoop-common-2.8.0.jar:na]  
at com.dremio.exec.util.ImpersonationUtil.createFileSystem(ImpersonationUtil.java:202) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.exec.util.ImpersonationUtil.createFileSystem(ImpersonationUtil.java:194) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.exec.store.dfs.FileSystemPlugin.getFS(FileSystemPlugin.java:249) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.exec.store.dfs.FileSystemStoragePlugin2.(FileSystemStoragePlugin2.java:92) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.plugins.s3.store.S3StoragePlugin2.(S3StoragePlugin2.java:34) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.plugins.s3.store.S3Plugin.getStoragePlugin2(S3Plugin.java:71) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.plugins.s3.store.S3Plugin.getStoragePlugin2(S3Plugin.java:45) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.exec.store.StoragePluginRegistryImpl.create(StoragePluginRegistryImpl.java:460) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.exec.store.StoragePluginRegistryImpl.createOrUpdate(StoragePluginRegistryImpl.java:288) ~[dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:124) ~[dremio-dac-backend-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:116) ~[dremio-dac-backend-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:81) ~[dremio-dac-backend-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source) ~[na:na]  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0\_151]  
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0\_151]  
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]  
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]  
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]  
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]  
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]  
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]  
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]  
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]  
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]  
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]  
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]  
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]  
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]  
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]  
at java.lang.Thread.run(Thread.java:748) [na:1.8.0\_151]  
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 6E73AAC8CA5049FB)  
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[aws-java-sdk-core-1.11.156.jar:na]  
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221) ~[aws-java-sdk-s3-1.11.156.jar:na]  
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168) ~[aws-java-sdk-s3-1.11.156.jar:na]  
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4162) ~[aws-java-sdk-s3-1.11.156.jar:na]  
at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:914) ~[aws-java-sdk-s3-1.11.156.jar:na]  
at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:920) ~[aws-java-sdk-s3-1.11.156.jar:na]  
at com.dremio.plugins.s3.store.S3FileSystem.initialize(S3FileSystem.java:110) ~[dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]  
… 69 common frames omitted

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [December 4, 2017, 7:07pm UTC](https://community.dremio.com/t/problem-adding-an-s3-source/514/7 "2017-12-04T19:07:20Z")

</div>

Hi @Jorge_Baez

Sorry did not realize it was not a public bucket. Let me get back to you on this shortly

Thanks  
@balaji.ramaswamy

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [December 5, 2017, 6:14pm UTC](https://community.dremio.com/t/problem-adding-an-s3-source/514/8 "2017-12-05T18:14:03Z")

</div>

HI @Jorge_Baez

One other thing, have you defined any list bucket policies under the IAM policy of the user trying to access this bucket?

Thanks,  
@balaji.ramaswamy

---

<div class="post-metadata">

**Author:** ![Jorge\_Baez](https://avatars.discourse-cdn.com/v4/letter/j/d78d45/32.png) [@Jorge\_Baez](https://community.dremio.com/u/Jorge_Baez)\
**Post date:** [December 5, 2017, 6:30pm UTC](https://community.dremio.com/t/problem-adding-an-s3-source/514/9 "2017-12-05T18:30:26Z")

</div>

Hi @balaji.ramaswamy

Yes, list/get/delete/put actions are enabled for the user, since this key/secret pair is currently being used from awscli and some custom python scripts, using boto. In both cases directories are listed, and files are uploaded/downloaded with the same key/secret pair configured in Dremio.

Just to be sure:  
The field “Name” is just a descriptive name for the source, and apart from the fields of the key and the secret, in the external buckets section, the name of the bucket should be provided in some special form? We tried with different variations of s3://bucketname and also just the bucketname, and none of them worked.