After upgrading a helm deployment of Dremio from 22.0 to 22.1, it fails to start. This appears to be due to a regression in the distributed storage feature when connecting to MinIO. In our deployment, Dremio’s distributed storage feature is configured with type aws and connected to MinIO (as an S3-equivalent service), with the extra properties below configured for compatibility:
<property>
<name>fs.s3a.endpoint</name>
<value>placeholder:9000</value>
</property>
<property>
<name>fs.s3a.path.style.access</name>
<description>Value has to be set to true.</description>
<value>true</value>
</property>
<property>
<name>dremio.s3.compat</name>
<description>Value has to be set to true.</description>
<value>true</value>
</property>
The error below is from the master’s logs. Notice that the bucket name is being prepended to the URL. This suggests that Dremio 22.1 is not honoring the fs.s3a.path.style.access setting.
2022-09-16 01:26:12,808 [start-__home] ERROR c.d.p.s3.store.S3FSHealthChecker - Error while listing S3 objects in /bucketName/pvdremio/distStorage/uploads
com.amazonaws.SdkClientException: Unable to execute HTTP request: bucketName.dev-systemlink-minio.foo-bar.svc.cluster.local: Name or service not known