Changing S3 distStorage Bucket Not Working

1

Hey all,

The Problem:

I am troubleshooting an issue nearly identical to the one in this older post here: Can't change s3 dist bucket
I need to change the bucket I am using for distributed storage, but when I do it is still trying to connect to the old bucket as shown in the Dremio logs.

What I Have Tried:

I have tried giving intentionally incorrect bucket names and paths like shown below to make it easy for me to search in the Dremio log to see if its actually getting picked up. Regardless of the value I give for the bucketName and path, it always uses the old bucket name and path. Changing the endpoint values does seem to take effect though, along with the credentials values. I have manually restarted the pods, helm upgraded using the updated values.yaml file, uninstalled dremio and reinstalled it, etc.

I am concerned that the bucket name and path are cached in the PVC’s and that I will need to wipe the PVC’s in order for my changes to take effect. This would delete a significant amount of work I have done in my Dremio environment.

My Configuration and Log:

values.yaml

distStorage:
  type: "aws"
  aws:
    bucketName: "TESTBUCKET"
    path: "/TESTPATH"
    authentication: "accessKeySecret"
    credentials:
      accessKey: "using-real-access-key-here"
      secret: "using-real-secret-here"
    extraProperties: |
      <property>
        <name>fs.s3a.endpoint</name>
        <value>using-real-endpoint-here</value>
      </property>
      <property>
        <name>fs.s3a.endpoint.region</name>
        <value>us-east-1</value>
      </property>
      <property>
        <name>fs.s3a.connection.ssl.enabled</name>
        <value>true</value>
      </property>

Error Log (the bucket “datamesh” shown here is the old bucket name):

2025-01-28 16:20:03.677	
java.io.IOException: Failure to create directory /datamesh/data/results_cache.
	2025-01-28 16:20:03.675	
java.io.IOException: Failure to create directory /datamesh/scratch.
	2025-01-28 16:20:03.673	
java.io.IOException: Failure to create directory /datamesh/accelerator.
	2025-01-28 16:20:03.672	
java.io.IOException: Failure to create directory /datamesh/downloads.
	2025-01-28 16:20:03.672	
java.io.IOException: Failure to create directory /datamesh/system_iceberg_tables.
	2025-01-28 16:20:03.657	
java.io.IOException: Failure to create directory /datamesh/metadata.
	2025-01-28 16:20:03.656	
java.io.IOException: Failure to create directory /datamesh/node_history.
	2025-01-28 16:18:05.387	
java.io.IOException: Failure to create directory /datamesh/node_history.
	2025-01-28 16:18:05.372	
java.io.IOException: Failure to create directory /datamesh/accelerator.
	2025-01-28 16:18:05.331	
java.io.IOException: Failure to create directory /datamesh/data/results_cache.
	2025-01-28 16:18:05.329	
java.io.IOException: Failure to create directory /datamesh/scratch.
	2025-01-28 16:18:05.327	
java.io.IOException: Failure to create directory /datamesh/system_iceberg_tables.
	2025-01-28 16:18:05.325	
java.io.IOException: Failure to create directory /datamesh/metadata.
	2025-01-28 16:18:05.325	
java.io.IOException: Failure to create directory /datamesh/downloads.
	2025-01-28 16:15:12.786	
java.io.IOException: Failure to create directory /datamesh/accelerator.
	2025-01-28 16:15:12.782	
java.io.IOException: Failure to create directory /datamesh/system_iceberg_tables.
	2025-01-28 16:15:12.781	
java.io.IOException: Failure to create directory /datamesh/metadata.
	2025-01-28 16:15:12.779	
java.io.IOException: Failure to create directory /datamesh/node_history.
	2025-01-28 16:15:12.778	
java.io.IOException: Failure to create directory /datamesh/data/results_cache.
	2025-01-28 16:15:12.774	
java.io.IOException: Failure to create directory /datamesh/downloads.
	2025-01-28 16:15:12.774	
java.io.IOException: Failure to create directory /datamesh/scratch.
2

For those in the future having the same issue: I was able to resolve this by updating the Dremio image version in the values.yaml file from 25.2 to 25.2.2 which happens to be the latest version at this time. I dont believe the image version itself resolved the issue based on the change notes, but rather the fact that the image changed at all.

3

@lewis133 Change to helm charts require redeploying them, did you do that initially?

4

Yes I did, had a call with Dave Anderson and he suggested changing the image version. That was the only thing that we tried that resolved this issue