Can't add s3 data source

1

I am using dremio-community-4.0.1-201909191652190301_211720e_1.noarch.rpm

I have dremio (single node) running aws ec2 (under company vpc), server is up and I am able to login to dremio UI and create postgres data source and run query.

When I try to create aws s3 data source with option “ec2 metadata”, it failed. UI showed "Failure while configuring source [rlin-test]. And server log showed: “Connect to sts.us-east-1.amazonaws.com:443 [sts.us-east-1.amazonaws.com/54.239.16.72] failed: connect timed out”.

I did add the required read policy to the role that was assigned to ec2. I am able to run the following commands by logging into ec2:
[ec2-user@ip-10-83-21-86 ~] aws s3 ls trhub1stg-cato --recursive [ec2-user@ip-10-83-21-86 ~] aws sts get-caller-identity

2

@rlin

Do you have Proxy setup, see below documentation link

Connecting through a Proxy Server

Thanks
@balaji.ramaswamy

3

I don’t have a proxy setup. When I login to ec2 and I am able to use the following command to view to read s3 files:
aws s3 ls —recursive
aws s3 cp …

4

And I tested it again by adding proxy host & port, same error

5

From ec2 instance, I su to user “dremio” and I am able to run the following

bash-4.2$ curl -i https://sts.us-east-1.amazonaws.com
HTTP/1.1 200 Connection established

HTTP/1.1 302 Found
x-amzn-RequestId: c0ff9544-e25a-11e9-87e4-615e53ca81b8
Location: https://aws.amazon.com/iam
Content-Length: 0
Date: Sun, 29 Sep 2019 01:45:01 GMT

6

add -Dhttp.proxyHost and -Dhttp.proxyPort to java options, still doesn’t work

image
image.png1111×159 16.7 KB

7

I searched the source code, I don’t think it is used for sts call. Is my understanding right?

|Property Name|Description|
|fs.s3a.proxy.host|Proxy host.|
|fs.s3a.proxy.port|Proxy port number.|
|fs.s3a.proxy.username|Username for authenticated connections, optional.|
|fs.s3a.proxy.password|Password for authenticated connections, optional.|

8

error was thrown from the following code.

image
image.png725×240 10.4 KB

line 193 comes from:
…
case EC2_METADATA_PROVIDER:
return InstanceProfileCredentialsProvider.create();
…

9

@rlin

Try adding this to your S3 source advanced tab-add property

fs.s3a.proxy.host
fs.s3a.proxy.port
fs.s3a.endpoint

10

It did not work. The error was caused by calling sts.

11

Hi @rlin

On the S3 source, under advanced settings, is “Enable asynchronous access when possible” is checked? if it is, can we try to disable and retry?

Thanks
@balaji.ramaswamy

12

not working either way. We are working with dremio team to find the root and solution.

13

For us, this same error occurred immediately upon updating from version 3.3.2-201908141640190085-d60145d to version 4.0.2-201910020123580864-a98a0b9. We’ve performed similar troubleshooting steps and have found the same issues as @rlin.

We’re also working with our support team to figure out why this is occurring.

14

@rlin, we enabled Compatibility Mode (Experimental) in the Advanced Options of our S3 Connection and it immediately started working again. It might be a fluke, but now we’re able to run queries.

There were some new fields added to the advanced options, most notably Encryption Key ARN. As we use several different encryption keys within that s3 connection, we can’t use that field. Perhaps the compatibility mode gets Dremio to ignore that field when making the API call? Just speculation.

15

@rlin

I was about to update the same. Try @kwcrook suggestion and update us. Enable ,“enable compatibility mode (experimental)”

Thanks
@balaji.ramaswamy

16

@kwcrook suggestion worked !!! Thank you both.

17

@rlin

Is your S3 a regular AWS S3 or something like Minio?

Thanks
@balaji.ramaswamy

18

We use Regular Aws s3

19

Hi @balaji.ramaswamy,
I am having a similar issue with Wasabi S3 buckets (like Minio).

@Ben kindly helped me, but I thought I’d loop you in in case you had seen this.
It appears the Dremio S3 layer is trying to authenticate against AWS rather than Wasabi. Is there a property or configuration that would force Dremio to authenticate against Wasabi?

Thanks,

Eric