S3-like storage for Distributed Storage

#1

Hi,

I’m following the documentation to stand up Dremio in an on-premise k8s kluster, using the helm chart dremio_v2. Since Dremio version 21 doesn’t support “local” as an alternative for distributed storage, I’m going with S3 since we have that available as an on-premise offering through StorageGrid. The settings are the same as when using Minio.

I’ve set up a bucket per the documentation and named it aptly ‘dremio’.

However, I’m getting the following error UnknownHostException: dremio.<redacted>.com.

To me it looks like Dremio is going for a domain-style access pattern which neither Minio nor StorageGrid supports. I was under the impression that S3-like interfaces was supported for distributed storage, since there is a special section for it in the documentation.

I’ve added all core-site.xml properties in the key distStorage.aws.extraProperties

Has anyone had any luck with using either Minio or StorageGrid or any other S3-like interfaces as a distributed storage? Or is it just the K8s version of Dremio that doesn’t fully support this?

Kind regards,
Jesper

#2

@ramworks both StorageGrid and MinIO should work fine with Dremio. Last I tested was with v20.0. Common mistakes are putting http(s):// prefix for the endpoint or missing out to put the port.

Here’s a sample values.yaml that work for me (only relevant sections):

#... Line #287

distStorage:
  type: "aws"

#... Line #344

  aws:
    bucketName: "dremio"
    path: "/"
    authentication: "accessKeySecret" 
    credentials:
      accessKey: "ACCESS_KEY"
      secret: "SECRET_KEY"

    extraProperties: |
      <property>
        <name>fs.s3a.endpoint</name>
        <value>175.1.2.3:9000</value>
      </property>
      <property>
        <name>fs.s3a.path.style.access</name>
        <value>true</value>
      </property>
      <property>
        <name>dremio.s3.compat</name>
        <value>true</value>
      </property>
      <property>
        <name>fs.s3a.connection.ssl.enabled</name>
        <value>false</value>
      </property>
2 Likes
#3

@lenoyjacob Our vaules.yaml files look identical in that regards. Only difference is that we have https for our StorageGrid endpoint. Here’s mine, for reference:

distStorage:
  type: "aws"
  aws:
    bucketName: "dremio"
    path: "/"
    authentication: "accessKeySecret"
    credentials:
      accessKey: "ACCESS_KEY"
      secret: "SECRET_KEY"
    extraProperties: |
      <property>
        <name>fs.s3a.endpoint</name>
        <value>storagegrid.endpoint.company.com</value>
      </property>
      <property>
        <name>fs.s3a.path.style.access</name>
        <value>true</value>
      </property>
      <property>
        <name>dremio.s3.compat</name>
        <value>true</value>
      </property>
      <property>
        <name>fs.s3a.connection.ssl.enabled</name>
        <value>true</value>
      </property>

The annoying thing is, I got it to work (yay!) when having the following image:

image: dremio/dremio-oss
imageTag: 20.1.0

But not any later tags/versions. Later versions always result in the error stated in the OP. Probably should file this as a bug somewhere. Anyhow, thanks for your help in confirming that at least my settings were correct!

Cheers!

#4

@ramworks Are you blocked on your upgrade to v21.x due to this?

#5

@balaji.ramaswamy Yes, unfortunately. Seems that neither versions of 21.x respects the additional properties fully.