Distributed Storage on openstack swift s3

#1

Hi,
I’m trying to use Openstack swift s3 compatibility storage for Distributed Storage.
But it doesn’t work.
Dremio version 4.9.1
Here my core-site.xml

<?xml version="1.0"?>
 <configuration>
 <property>
     <name>fs.dremioS3.impl</name>
     <value>com.dremio.plugins.s3.store.S3FileSystem</value>
 </property>
 <property>
     <name>fs.s3a.access.key</name>
     <value>XXXX</value>
 </property>
 <property>
     <name>fs.s3a.secret.key</name>
     <value>XXXX</value>
 </property>
 <property>
     <name>fs.s3a.aws.credentials.provider</name>
     <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
 </property>
 <property>
     <name>fs.s3a.endpoint</name>
     <value>s3.ru-1a.selcdn.ru</value>
 </property>
 <property>
     <name>fs.s3a.path.style.access</name>
     <value>true</value>
 </property>
 <property>
     <name>dremio.s3.compat</name>
     <value>true</value>
 </property>
 <property>
     <name>fs.s3a.connection.ssl.enabled</name>
     <value>SSL_ENABLED</value>
 </property>
 </configuration>

But, I got ERROR:

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your AWS secret access key and signing method. For more information, see REST Authentication and SOAP Authentication for details. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: acc83468-2675-47af-97c6-6006fa41ca21; S3 Extended Request ID: null)

But my auth credentials is correct.

Where is my problem?
Can Dremio Distributed Storage work with openstack swift s3?
Has anyone had a similar setup experience?

Thanks!

#2

I changed option in my S3 Cloud Storage and got new problem:

[start-__home] WARN c.d.e.catalog.ManagedStoragePlugin - Error starting new source: __home
java.lang.NullPointerException: null uri host.
WARN c.d.e.catalog.ManagedStoragePlugin - Error starting new source: __accelerator
java.lang.NullPointerException: null uri host.

I’m starting new installation Dremio and had the same problem

#3

@alexdi

Dremio currently does not support Swift, Have you tried to enable “Enable compatibility mode (experimental)” under advanced in the S3 source setting?

#4

Hi, thanks for answer
I got next error:

image
image3346×670 133 KB

What does it mean?
Here my config:
image
image1692×1780 189 KB

image
image1690×1782 191 KB

And where can i set endpoint (IP address) and region of my Swift S3 provider?

#5

@alexdi

Have you tried to add via advanced tab under add property?

#6

Hi, Yes I did

image
image1238×282 12.5 KB

But, I got next error:

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The authorization header is malformed; the region ‘selcdn’ is wrong; expecting ‘ru-1’ (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: 6e4cbb62-6da5-40dc-abbd-bf9b8712ff71; S3 Extended Request ID: null)

What is the property name for set region?

#7

@alexdi

Your core-site has a different end point? “s3.ru-1a.selcdn.ru”

#8

Yes, but if I set s3.ru-1a.selcdn.ru I got error which i described in the top of this topic “null uri host.”
Thanks for help, I think dremio doesn’t support Openstack swift s3 and i connected dremio to self hosting MinIO.

#9

@alexdi I can confirm that we have the same issue. We are able to connect OpenStack Swift in Compatibility mode, however we are not able to use it as Distributed Cached Store for Reflexions, or even reading parquet files hosted in the object storage.

We have already provided a solution to Dremio Team since Dec 2020, but the issue still persists.

We have also tested this issue with version15.0, sitll same results:

S3Exception: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘BHS’ (Service: S3, Status Code: 400, Request ID: txd785d13046bd40eca97ca-006072f251)

#10

@asakmedops

To use distributed store, you would have to add the compat. flag in core-site.xml. Have you tried that?

<configuration>
<name>dremio.s3.compat</name>
<description>Compatiable mode, not calling AWS STS</description>
<value>true</value>
</property>
</configuration>
#11

If you see my first post, I added this flag, but dremio didn’t connect to OpenStack Swift

#12

@alexdi

S if I understand it right, source works but distributed store does not work?

#13

@balaji.ramaswamy

Distributed store works for non parquet files. For example a csv file can be read without any problem.
Just trying to read a parquet file straight from the S3 Storage , it fails with the same error above.
We already reported and did the troubleshoot here