Distributed Storage on openstack swift s3

#1

Hi,
I’m trying to use Openstack swift s3 compatibility storage for Distributed Storage.
But it doesn’t work.
Dremio version 4.9.1
Here my core-site.xml

<?xml version="1.0"?>
 <configuration>
 <property>
     <name>fs.dremioS3.impl</name>
     <value>com.dremio.plugins.s3.store.S3FileSystem</value>
 </property>
 <property>
     <name>fs.s3a.access.key</name>
     <value>XXXX</value>
 </property>
 <property>
     <name>fs.s3a.secret.key</name>
     <value>XXXX</value>
 </property>
 <property>
     <name>fs.s3a.aws.credentials.provider</name>
     <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
 </property>
 <property>
     <name>fs.s3a.endpoint</name>
     <value>s3.ru-1a.selcdn.ru</value>
 </property>
 <property>
     <name>fs.s3a.path.style.access</name>
     <value>true</value>
 </property>
 <property>
     <name>dremio.s3.compat</name>
     <value>true</value>
 </property>
 <property>
     <name>fs.s3a.connection.ssl.enabled</name>
     <value>SSL_ENABLED</value>
 </property>
 </configuration>

But, I got ERROR:

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your AWS secret access key and signing method. For more information, see REST Authentication and SOAP Authentication for details. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: acc83468-2675-47af-97c6-6006fa41ca21; S3 Extended Request ID: null)

But my auth credentials is correct.

Where is my problem?
Can Dremio Distributed Storage work with openstack swift s3?
Has anyone had a similar setup experience?

Thanks!

#2

I changed option in my S3 Cloud Storage and got new problem:

[start-__home] WARN c.d.e.catalog.ManagedStoragePlugin - Error starting new source: __home
java.lang.NullPointerException: null uri host.
WARN c.d.e.catalog.ManagedStoragePlugin - Error starting new source: __accelerator
java.lang.NullPointerException: null uri host.

I’m starting new installation Dremio and had the same problem

#3

@alexdi

Dremio currently does not support Swift, Have you tried to enable “Enable compatibility mode (experimental)” under advanced in the S3 source setting?

#4

Hi, thanks for answer
I got next error:

image
image3346×670 133 KB

What does it mean?
Here my config:
image
image1692×1780 189 KB

image
image1690×1782 191 KB

And where can i set endpoint (IP address) and region of my Swift S3 provider?

#5

@alexdi

Have you tried to add via advanced tab under add property?

#6

Hi, Yes I did

image
image1238×282 12.5 KB

But, I got next error:

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The authorization header is malformed; the region ‘selcdn’ is wrong; expecting ‘ru-1’ (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: 6e4cbb62-6da5-40dc-abbd-bf9b8712ff71; S3 Extended Request ID: null)

What is the property name for set region?

#7

@alexdi

Your core-site has a different end point? “s3.ru-1a.selcdn.ru”

#8

Yes, but if I set s3.ru-1a.selcdn.ru I got error which i described in the top of this topic “null uri host.”
Thanks for help, I think dremio doesn’t support Openstack swift s3 and i connected dremio to self hosting MinIO.