Connecting S3-compatible storage cannot run sql in executor node due to path style access

1

Dremio Version: 24.0.0
ENV: microk8s

When connecting to S3-compatible storage(Ali OSS which only support virtual hosted accessing) as data source, there is no problem in the webui and coordinator. Files can be loaded in the webui. However, when executing sql, there will be error in executor node as below(“Please use virtual hosted style to access.”).

image
image1892×522 35.4 KB

I have already set fs.s3a.path.style.access to false, but it seems that the config can only affect coordinator node, executor node running sqls still use path style access. Is there any way to solve the problem? Thanks a lot for help!

ERROR:

PERMISSION ERROR: Access was denied by S3

SqlOperatorImpl TABLE_FUNCTION
Location 1:3:4
Fragment 1:0

[Error Id: c4fb5461-0c28-4a56-b94b-166bfe0bb053 on dremio-executor-1.dremio-cluster-pod.dremio.svc.cluster.local:0]

(java.util.concurrent.CompletionException) software.amazon.awssdk.services.s3.model.S3Exception: Please use virtual hosted style to access. (Service: S3, Status Code: 403, Request ID: 64367DF6FF584737324F3D4B, Extended Request ID: null)
software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException():60
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0():51
java.util.concurrent.CompletableFuture.uniHandle():836
java.util.concurrent.CompletableFuture$UniHandle.tryFire():811

2

9c37a841-6165-418a-9f32-2c7a0d8dd999.zip (16.4 KB)

this is the profile for the sql~

3

tried to add fs.s3a.path.style.access=false in extraStartParams:. It not works :smiling_face_with_tear:

4

Hi @hxj8059

Can you try checking the below in advanced settings and see if it works? Enable compatibility mode

5

image
image1856×374 29.4 KB

Thanks for your reply. The compatibility mode was enabled. What is more interesting is that I can load csv or excel file in the same oss data source. Only parquet file cannot be loaded. Is this because when executor nodes read parquet files, the “fs.s3a.path.style.access=false” config does not take effect?

6

Hi @balaji.ramaswamy,

I enabled the compatibility mode before, and turn it off now. The problem is still there. Aliyun OSS in fact is compatible with S3 but it only supports virtual hosted access instead of path style access. It seems that the access config does not take effect in executor nodes.

7

Virtual-hosted style is the default one.
What’s the content of your code-site.xml?

ps might need to turn on debug logging for S3 to isolate which request in particular is failing here

8

I haven’t edit the core-site.xml because i thought that edit value.yaml will change the config in core-site.xml. Trying to turn on the logging for S3

9

I guess the problem should be related to reading parquet and access style since the error occurs during ParquetScan

10

Did you solve this problem?

11

@colagy Kindly provide your profile as often the top level error might be the same while the actual stack will gives us Caused By