S3 (Compat Mode) errors while getting data from reflection

1

Here is the error we get when we try to read Data from Reflection:

software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the
region ‘us-east-1’ is wrong; expecting ‘bhs’ (Service: S3, Status Code: 400, Request ID:
tx22fe877545b84c10af5fe-005fce53e4)

We are using compatibility mode of S3. Everything else works fine except when trying to use reflections.

We are using Dremio(v 11) with Kubernetes with following additional properties with AWS distributed Storage:

  <property>
  <name>dremio.s3.compat</name>
  <value>true</value>
  </property>
  <property>
    <name>fs.s3a.endpoint</name>
    <value>s3.bhs.cloud.ovh.net</value>
  </property>
2

@asakmedops

For the reflections, did you add the above parameters to core-site.xml under the configs folder?

3

Initially We have added the parameters under extraParams ( distStorage.aws).
With your comments, We Tested with configs under core-site.xml, but we still have the same errors

4

We also have the same errors while trying to read parquet files from S3 ( S3 Compatible).
We added a parquet file as a Table from Dremio, but we were Unable to read it.

We are able to read files ( csv, excel …) in S3 (S3 Compatible) . I think the issue is just while trying to read parquet files from S3 (S3 Compat).

Looks this PR Use config fs.s3a.path.style.access value for S3 Access is addressing the issue, but was never merged

5

@asakmedops

Wondering if the server.log has a detailed stack, are you ok to send us the complete server.log?

6

Here are the logs . This is when I try to read a table located on S3 that has already been formatted to parquet. dremio-logs-s3.logs.txt.zip (2.8 KB)

7

Hi, I am the developer of the patch. It is actually fixed my problem and we can use S3 as storage for reflections. I could update the PR to be compatible with the latest Dremio master, but seems there is no interest to merge it from the Dremio team.

8

@deem0n
This dremio-w-minio-distributed-store-failures is a related issue as well. Looks like an issue that everyone trying to use anything different then S3 will have. @balaji.ramaswamy is Dremio Team aware of this bug and the PR to fix it?

9

Thanks for the update @deem0n and @asakmedops, will forward to the relevant team

10

In fact, the error is related to reading any parquet file located on any S3 compatible Server.
We are able to read any other file type (csv, excel,…) but reading parquet fails:

here are more logs to help fix the issue:
dremio-error-read-parquet-s3.zip (3.6 KB)

11

Any progress on this issue? I am experiencing the same problem.

Solution: I fixed it in the meanwhile by adding to the core-site.xml the region

<property>
  <name>dremio.s3.region</name>
  <value>**REGION**</value>
</property>