# Can't add s3 data source **URL:** \ **Category:** Uncategorized\ **Created:** [September 27, 2019, 7:43pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199 "2019-09-27T19:43:29Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![rlin](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rlin](https://community.dremio.com/u/rlin)\ **Post date:** [September 27, 2019, 7:43pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/1 "2019-09-27T19:43:29Z")
I am using dremio-community-4.0.1-201909191652190301\_211720e\_1.noarch.rpm I have dremio (single node) running aws ec2 (under company vpc), server is up and I am able to login to dremio UI and create postgres data source and run query. When I try to create aws s3 data source with option “ec2 metadata”, it failed. UI showed "Failure while configuring source [rlin-test]. And server log showed: “Connect to [sts.us-east-1.amazonaws.com:443](http://sts.us-east-1.amazonaws.com:443) [[sts.us-east-1.amazonaws.com/54.239.16.72](http://sts.us-east-1.amazonaws.com/54.239.16.72)] failed: connect timed out”. I did add the required read policy to the role that was assigned to ec2. I am able to run the following commands by logging into ec2: [ec2-user@ip-10-83-21-86 ~] aws s3 ls trhub1stg-cato --recursive [ec2-user@ip-10-83-21-86 ~] aws sts get-caller-identity ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [September 28, 2019, 7:56pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/2 "2019-09-28T19:56:57Z")
@rlin Do you have Proxy setup, see below documentation link [Connecting through a Proxy Server](https://docs.dremio.com/data-sources/s3.html#connecting-through-a-proxy-server) Thanks @balaji.ramaswamy ---
**Author:** ![rlin](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rlin](https://community.dremio.com/u/rlin)\ **Post date:** [September 28, 2019, 9:19pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/3 "2019-09-28T21:19:20Z")
I don’t have a proxy setup. When I login to ec2 and I am able to use the following command to view to read s3 files: aws s3 ls —recursive aws s3 cp … ---
**Author:** ![rlin](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rlin](https://community.dremio.com/u/rlin)\ **Post date:** [September 29, 2019, 12:09am UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/4 "2019-09-29T00:09:12Z")
And I tested it again by adding proxy host & port, same error ---
**Author:** ![rlin](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rlin](https://community.dremio.com/u/rlin)\ **Post date:** [September 29, 2019, 1:47am UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/5 "2019-09-29T01:47:36Z")
From ec2 instance, I su to user “dremio” and I am able to run the following bash-4.2$ curl -i [https://sts.us-east-1.amazonaws.com](https://sts.us-east-1.amazonaws.com) HTTP/1.1 200 Connection established HTTP/1.1 302 Found x-amzn-RequestId: c0ff9544-e25a-11e9-87e4-615e53ca81b8 Location: [https://aws.amazon.com/iam](https://aws.amazon.com/iam) Content-Length: 0 Date: Sun, 29 Sep 2019 01:45:01 GMT ---
**Author:** ![rlin](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rlin](https://community.dremio.com/u/rlin)\ **Post date:** [September 29, 2019, 4:05pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/6 "2019-09-29T16:05:46Z")
add -Dhttp.proxyHost and -Dhttp.proxyPort to java options, still doesn’t work ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/3/30442b38fff0e475a690eb5a6df30a7bb5790944.png) ---
**Author:** ![rlin](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rlin](https://community.dremio.com/u/rlin)\ **Post date:** [September 29, 2019, 4:12pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/7 "2019-09-29T16:12:23Z")
I searched the source code, I don’t think it is used for sts call. Is my understanding right? |Property Name|Description| |fs.s3a.proxy.host|Proxy host.| |fs.s3a.proxy.port|Proxy port number.| |fs.s3a.proxy.username|Username for authenticated connections, optional.| |fs.s3a.proxy.password|Password for authenticated connections, optional.| ---
**Author:** ![rlin](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rlin](https://community.dremio.com/u/rlin)\ **Post date:** [September 29, 2019, 4:32pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/8 "2019-09-29T16:32:48Z")
error was thrown from the following code. ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/1/1e8961cf8852c879a775dc1c8a4ccd4bb599e744.png) line 193 comes from: … case EC2\_METADATA\_PROVIDER: return InstanceProfileCredentialsProvider.create(); … ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [September 30, 2019, 5:37am UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/9 "2019-09-30T05:37:06Z")
@rlin Try adding this to your S3 source advanced tab-add property fs.s3a.proxy.host fs.s3a.proxy.port fs.s3a.endpoint ---
**Author:** ![rlin](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rlin](https://community.dremio.com/u/rlin)\ **Post date:** [September 30, 2019, 8:20pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/10 "2019-09-30T20:20:16Z")
It did not work. The error was caused by calling sts. ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [October 8, 2019, 3:19pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/11 "2019-10-08T15:19:32Z")
Hi @rlin On the S3 source, under advanced settings, is “Enable asynchronous access when possible” is checked? if it is, can we try to disable and retry? Thanks @balaji.ramaswamy ---
**Author:** ![rlin](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rlin](https://community.dremio.com/u/rlin)\ **Post date:** [October 8, 2019, 6:03pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/12 "2019-10-08T18:03:57Z")
not working either way. We are working with dremio team to find the root and solution. ---
**Author:** ![kwcrook](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kwcrook/32/2079_2.png) [@kwcrook](https://community.dremio.com/u/kwcrook)\ **Post date:** [October 10, 2019, 7:15pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/13 "2019-10-10T19:15:36Z")
For us, this same error occurred immediately upon updating from version 3.3.2-201908141640190085-d60145d to version 4.0.2-201910020123580864-a98a0b9. We’ve performed similar troubleshooting steps and have found the same issues as @rlin. We’re also working with our support team to figure out why this is occurring. ---
**Author:** ![kwcrook](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kwcrook/32/2079_2.png) [@kwcrook](https://community.dremio.com/u/kwcrook)\ **Post date:** [October 10, 2019, 7:36pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/14 "2019-10-10T19:36:43Z")
@rlin, we enabled Compatibility Mode (Experimental) in the Advanced Options of our S3 Connection and it immediately started working again. It might be a fluke, but now we’re able to run queries. There were some new fields added to the advanced options, most notably `Encryption Key ARN`. As we use several different encryption keys within that s3 connection, we can’t use that field. Perhaps the compatibility mode gets Dremio to ignore that field when making the API call? Just speculation. ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [October 10, 2019, 8:29pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/15 "2019-10-10T20:29:03Z")
@rlin I was about to update the same. Try @kwcrook suggestion and update us. Enable ,“enable compatibility mode (experimental)” Thanks @balaji.ramaswamy ---
**Author:** ![rlin](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rlin](https://community.dremio.com/u/rlin)\ **Post date:** [October 10, 2019, 11:24pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/16 "2019-10-10T23:24:41Z")
@kwcrook suggestion worked !!! Thank you both. ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [October 11, 2019, 5:30pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/17 "2019-10-11T17:30:58Z")
@rlin Is your S3 a regular AWS S3 or something like Minio? Thanks @balaji.ramaswamy ---
**Author:** ![rlin](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rlin](https://community.dremio.com/u/rlin)\ **Post date:** [October 12, 2019, 11:22am UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/18 "2019-10-12T11:22:20Z")
We use Regular Aws s3 ---
**Author:** ![eellsworth](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/eellsworth/32/2643_2.png) [@eellsworth](https://community.dremio.com/u/eellsworth)\ **Post date:** [December 21, 2020, 6:00pm UTC](https://community.dremio.com/t/cant-add-s3-data-source/4199/19 "2020-12-21T18:00:55Z")
Hi @balaji.ramaswamy, I am having a similar issue with Wasabi S3 buckets (like Minio). > [@S3 bucket from non-Amazon provider](https://community.dremio.com/t/s3-bucket-from-non-amazon-provider/6755/3): > > Ben, Thanks so much for this advice! Unfortunately I can’t seem to get Dremio to connect to the wasabi host ([s3.wasabisys.com](http://s3.wasabisys.com)). Here are the parameters I used: The exceptions sometimes show Dremio attempting to connect to [sts.amazonaws.com](http://sts.amazonaws.com), and other times don’t list host names - they never show traffic to [s3.wasabisys.com](http://s3.wasabisys.com) at com.dremio.plugins.s3.store.STSCredentialProviderV1.getCredentials(STSCredentialProviderV1.java:71) at org.apache.hadoop.fs.s3a.AWSCredentialProvid… @Ben kindly helped me, but I thought I’d loop you in in case you had seen this. It appears the Dremio S3 layer is trying to authenticate against AWS rather than Wasabi. Is there a property or configuration that would force Dremio to authenticate against Wasabi? Thanks, Eric