Cannot access public S3 bucket

1

Hello Dremio team,

I am a big fan of your work and product! I am having issues using Dremio though with a public S3 bucket of mine. I created this bucket with public access and has an Access Control List for Everyone to “List Objects” and “Read Bucket Permissions”. This account has other bucket that are private as well.

I get the following error:
java.lang.RuntimeException: java.io.IOException: Unable to retrieve bucket named “xyz”

My configuration has a name and “https://s3.amazonaws.com/xyz” listed under External Buckets and nothing else.

The documentation has only one slide on S3 that is very cryptic in terms of configuration on the Dremio side:
https://docs.dremio.com/data-sources/s3.html

Thank you in advance for your guidance,
George Zoto

2

@George_Zoto thank you fr the kind words. It might be that you are putting the full bucket URL – have you tried just with the bucket name?

Here is a sample S3 source where we connect to our own sample public bucket:

image

Also, are there any other details in server.log?

3

Hello @can,

Thank your for your insight, I was using the full https url. By updating the external bucket name with just the s3 bucket name I now get:

com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 86426831BBB1DAFB), S3 Extended Request ID:

And here are my permission settings for this bucket:

S3%201
S3 1.png1932×284 21.8 KB

S3%202
S3 2.png2536×356 19.7 KB

Am I missing something here?

Thank you,
George

4

Am I missing something simple here @can?

Thank you again,
George

5

What do you have under the “Bucket Policy” tab under “Permissions” ?
Maybe try setting it similar to below

{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Principal”: “",
“Action”: [
“s3:ListBucket”,
“s3:GetBucketLocation”
],
“Resource”: “arn:aws:s3:::samples.bucket”
},
{
“Sid”: “MakeItPublic”,
“Effect”: “Allow”,
“Principal”: "”,
“Action”: “s3:GetObject”,
“Resource”: “arn:aws:s3:::samples.bucket/*”
}
]
}

1 Like
6

Thank you @Anthony, it worked :slight_smile:

Just to understand though, ACLs are not enough to allow Dremio access to a public S3 bucket?
Do you need a bucket policy no matter what?

Also, in my case Principal field was not allowed to be empty so here is the revised policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": "arn:aws:s3:::mybucket"
},
{
"Sid": "MakeItPublic",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3::: mybucket/*"
}
]
}