Problems accessing S3 when running Dremio in a private subnet for executors

1

I using kubernetes cluster with one master and two slaves, i am accessing s3 bucket using "IAM role and policies and for slaves, in kubernetes slaves running 3 executors, in which i am facing problem with 1 executor with S3 access denied error,

Please find the following error

2021-11-09 12:28:45,623 [start-__home] INFO c.d.plugins.s3.store.S3FileSystem - User Error Occurred [ErrorId: 2411b7d8-c90e-4595-a1ae-072e5c31060d]
com.dremio.common.exceptions.UserException: Access was denied by S3
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:885)
at com.dremio.plugins.s3.store.S3FileSystem$BucketCreator$1.create(S3FileSystem.java:477)
at com.dremio.plugins.util.ContainerFileSystem$FileSystemSupplier.get(ContainerFileSystem.java:245)
at com.dremio.plugins.util.ContainerFileSystem$ContainerHolder.fs(ContainerFileSystem.java:203)
at com.dremio.plugins.util.ContainerFileSystem.mkdirs(ContainerFileSystem.java:475)
at com.dremio.exec.hadoop.HadoopFileSystem.mkdirs(HadoopFileSystem.java:279)
at com.dremio.dac.homefiles.HomeFileSystemStoragePlugin.start(HomeFileSystemStoragePlugin.java:105)
at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$1(ManagedStoragePlugin.java:546)
at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$3(ManagedStoragePlugin.java:614)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: AWGZ8S635GAWG5B9; S3 Extended Request ID: N2aIi9ztBk2I1QHtNAtL7X+eGpuodpFPmQxrYBLlVnGcAROIed2syMyJ6xO1n+C+DdhpGSWOx48=; Proxy: null)

kindly check and advise us, thanks.

2

Do you have cloudtrails logs? If you do, please look at the detailed event in cloudtrail and paste full event payload here

3

@shyambh_1 What do you exactly mean by “in kubernetes slaves running 3 executors, in which i am facing problem with 1 executor”, so if the request goes to another executor the query works?

4

Dremio_log.zip (40.9 KB)

Dremio-master pod is failing with s3 access denied and please find below error log details

In dremio_log.zip file for your reference,

Please find below crashloopbackoff dremio-Master-0 pod and pod describe details in dremio2.docx file

Please find below git link for dremio-master-executor#

dremio-cloud-tools/dremio-master.yaml at master · dremio/dremio-cloud-tools (github.com)

5

@shyambh_1 Are you able to look up the error message for the extended request ID from AWS support?

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 02K4CR0B6RDZWKR5; S3 Extended Request ID: BNPvQ6P1dr/rB7jsfp9VhwSoFV6S53aC1NbMJH3E+1ffe+uXgUDM25IqcfRZL2ksLR78y7paFSo=)

Also have we followed the below doc on giving the right privileges?

https://docs.dremio.com/data-sources/s3/