Dremio with Hive and S3

Dremio University
1

Hey Dremio folks!

I saw other posts with similar problems but their solutions seem not to work for me.

Recently I started to play with Dremio Community Edition, deployed on Kubernetes.

Build
24.2.6-202311250456170399-68acbe47
Edition
Community Edition
Build Time
11/25/2023 05:02:17
Change Hash
68acbe47566ebb7424d179776f7dc15fc6b81bdf
Change Time
11/25/2023 02:34:32

I’m trying to connect to our Hive Catalog to read Iceberg tables located in S3.
I’ve added a Hive 3 Source with the following Connection Properties, on Advanced Options:

fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
fs.s3a.access.key=xxx
fs.s3a.secret.key=yyy
dremio.s3.compat=true

The connection seems to work. On /source/ I can see all s3 folders listed. I can even see the metadata for some tables

image

My problem is when trying to query that table. I’m seeing this error:

AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 490K6Z4K365Q2P6H; S3 Extended Request ID: w0m56lkDQDKlodR4t/W6Svsk0XODaskLKr58e8qtFtkcMTKWKN7Ug1o7bGsmEerigOXj+Zu2knM=; Proxy: null)

Not sure what to do, as Dremio can read the folders and table metadata. Any ideas?

Thank you!!

2

Hi @kleinkauff,

Looks like the credentials are invalid, can you double check that’s not the case? Maybe they were valid at the time the source was added.
Also, you mentioned that the solutions for similar problems didn’t work for you. What were those solutions? If you can cross-reference them, that would be great.

Thanks, Bogdan

3

Hey, thank you so much for trying to help me.

I think the key is ok, this is the actions that are configured.

actions = [
  "s3:ListBucket",
  "s3:GetObject*",
  "s3:PutObject*",
  "s3:DeleteObject",
  "s3:PutBucketPublicAccessBlock",
  "s3:CreateBucket",
  "s3:GetBucketAcl",
  "s3:GetBucketLocation"
]

Strange thing is if I try other keys that do not have some of the permissions I see a different kind of message, like

image
image1256×125 34.5 KB

Some of other suggestions:
Enable compact mode - Connect Dremio to Iceberg Hive Catalog located at S3 - #3 by lenoyjacob

Compat mode in a different setting? - Connecting S3 bucket via ec2 metadata - #2 by balaji.ramaswamy - Not sure if this could help me

4

I think you’re missing s3:ListAllMyBuckets

5

@lenoyjacob Worked.

@bogdan.coman @lenoyjacob Thank you so much folks!