Dremio fails to start after upgrading to 22.1

1

Dremio 22.1 seems to have introduced a regression which prevents distributed storage from being hooked up to MinIO. Upon upgrading from 22.0 to 22.1, Dremio fails to start. We set the configuration option for forcing path style access as documented in Configuring Amazon S3 for MinIO access, but this seems to be ignored in 22.1, as the error from the log suggests that the bucket name is being incorrectly prepended to the hostname:

com.amazonaws.SdkClientException: Unable to execute HTTP request: bucket_name.<redacted_hostname>.

Caused by: java.net.UnknownHostException: bucket_name.<redacted_hostname>: Name or service not known

This bit of configuration is meant to prevent the bucket name from getting prepended, and it does in 22.0, but it isn’t honored in 22.1:

<property>
  <name>fs.s3a.path.style.access</name>
  <description>Value has to be set to true.</description>
  <value>true</value>
</property>

Here’s a full snippet of the distributed storage helm configuration to reproduce this:

distStorage:
  type: "aws"
  aws:
    bucketName: bucket_name
    path: "/dremio/distStorage"
    authentication: "accessKeySecret"
    credentials:
      accessKey: "<REDACTED"
      secret: "<REDACTED"
    extraProperties: |
      <property>
        <name>fs.s3a.endpoint</name>
        <value>hostname.local:9000</value>
      </property>
      <property>
        <name>fs.s3a.path.style.access</name>
        <description>Value has to be set to true.</description>
        <value>true</value>
      </property>
      <property>
        <name>dremio.s3.compat</name>
        <description>Value has to be set to true.</description>
        <value>true</value>
      </property>
      <property>
        <name>fs.s3a.connection.ssl.enabled</name>
        <description>Value can either be true or false, set to true to use SSL with a secure Minio server.</description>
        <value>false</value>
      </property>
2

This is addressed in 22.1.2 that’ll be out shortly.
22.1.0 does not have this issue if you need a version to fallback to in the meantime.
Using IP address instead of hostname for an endpoint is a workaround you could use as well.

2 Likes
3

Awesome, thank you for the quick update. Do you know when the GitHub repo will be updated to 22.1.2? We build our own image off of master.