Reflections fail when AWS account is under KMS SCP

1

Hello,

Our Infra team policy for AWS is to have all accounts under SCP to have KMS encryption.

When this is enforced, reflections fail with the following message:

AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: SY59BXH8RD9EY0FR; S3 Extended Request ID: 589rASZHuD0VVAVABHHxT70OzIVhS4KrbENC4EXL1aTn8H0HbNCa6Y+/mIK7JM7fVRc06KiPLew=; Proxy: null)

I have tried to add the KMS key encryption to the core-site.xml as below (actual values changed obviously) as per Dremio

but to no avail. the Reflections keep failing. Any pointers as to what needs to be done? Please help @balaji.ramaswamy / @Rafay

"
property>
fs.s3a.server-side-encryption-algorithm
SSE-KMS
"

"
fs.s3a.server-side-encryption.key
arn:aws:kms:eu-west-2:123456789:key/xxxxxb01-xxxa-xx9-b876-097ab8f5860b

"

(FYI - The post seems to not print out the actual literal copy paste as it is hiding the “property” open and close tag around the two settings.)

2

Hey there–

I have two quick questions for you. First, are you using the Enterprise version of Dremio? I believe this is the only one that supports the use of KMS encryption.

Second, have you verified that the core-site.xml is on all nodes (if you have multiple nodes)?

If you have checked both of these things, have you validated that you all the AWS s3 calls listed on the docs are available to your account that is configured for access? A 403 from AWS to me usually means there is a permission issue if all other bits of the configuration are correct.

Dan

3

Thanks for you prompt reply @danh .

We are on the AWS community edition v 24. So i guess, straight off that means KMS encryption is not supported.

We are using a single node instance, but we do have the customization.sh script etc. run so that it can copy to all nodes, if we choose to, but we think that option is unlikely.

I was looking at the distributed config page because that was the only place KMS encryption was mentioned. I was basically wanting use KMS encryption of the S3 bucket where the reflections get built. But it looks like this option does not exist for the community edition.