S3 Connection via EC2 Metadata *or* Temporary STS Credentials

Dremio University
1

Hey folks,

I’m in the following situation - we have an EC2 instance that already has access to s3 buckets via ec2 metadata, i.e. any type of aws cli command to access the s3 buckets works: aws s3 cp <s3> ./ and so on. We are limited to only using ec2 metadata, since our sts token strategy requires providing not only access key/secret but also a secret token.

Unfortunately, when trying to connect via dremio with ec2 metadata, the connection returns 403 forbidden. When I check the logs, I get similar output without a resolution strategy.

What’s the best way to continue debugging this? I have tried with the beta compatibility mode feature turned on and turned off, because I found that in previous threads.

2

Normally if you only use EC2 metadata to connect to S3, it is just ok. I do that too currently. Do you have S3 access log or cloudtrail logs for S3? If you do, please look at Dremio logs for eventID, then query that eventID in cloudtrail logs to get more information about the principal was doing the request

3

@lfy

Have you tried to check “Enable compatibility mode (experimental)” and tried again, should be under advanced options

4

Yes, I have tried with compatibility mode enabled as well as disabled.

5

@lfy Is there a a extended request ID printed in the logs? Have you tried to access the same bucket fron the Dremio coordinator’s command line?

6

Hi Balaji - can you show me where I could learn more about the Dremio coordinator command line? I am not sure where to find the documentation around that.

7

Judging by the CloudTrail logs, the issue is because dremio is attempting to call ‘ListBuckets’ which our ec2 metadata permissions don’t allow - we only provide permissions to access the bucket itself. Is there away to disable the call to listBuckets?

8

@lfy listbucket is currently a required privilege

https://docs.dremio.com/data-sources/s3/#sample-iam-policy-for-accessing-s3