Hey all,
The Problem:
I am troubleshooting an issue nearly identical to the one in this older post here: Can't change s3 dist bucket
I need to change the bucket I am using for distributed storage, but when I do it is still trying to connect to the old bucket as shown in the Dremio logs.
What I Have Tried:
I have tried giving intentionally incorrect bucket names and paths like shown below to make it easy for me to search in the Dremio log to see if its actually getting picked up. Regardless of the value I give for the bucketName and path, it always uses the old bucket name and path. Changing the endpoint values does seem to take effect though, along with the credentials values. I have manually restarted the pods, helm upgraded using the updated values.yaml file, uninstalled dremio and reinstalled it, etc.
I am concerned that the bucket name and path are cached in the PVC’s and that I will need to wipe the PVC’s in order for my changes to take effect. This would delete a significant amount of work I have done in my Dremio environment.
My Configuration and Log:
values.yaml
distStorage:
type: "aws"
aws:
bucketName: "TESTBUCKET"
path: "/TESTPATH"
authentication: "accessKeySecret"
credentials:
accessKey: "using-real-access-key-here"
secret: "using-real-secret-here"
extraProperties: |
<property>
<name>fs.s3a.endpoint</name>
<value>using-real-endpoint-here</value>
</property>
<property>
<name>fs.s3a.endpoint.region</name>
<value>us-east-1</value>
</property>
<property>
<name>fs.s3a.connection.ssl.enabled</name>
<value>true</value>
</property>
Error Log (the bucket “datamesh” shown here is the old bucket name):
2025-01-28 16:20:03.677
java.io.IOException: Failure to create directory /datamesh/data/results_cache.
2025-01-28 16:20:03.675
java.io.IOException: Failure to create directory /datamesh/scratch.
2025-01-28 16:20:03.673
java.io.IOException: Failure to create directory /datamesh/accelerator.
2025-01-28 16:20:03.672
java.io.IOException: Failure to create directory /datamesh/downloads.
2025-01-28 16:20:03.672
java.io.IOException: Failure to create directory /datamesh/system_iceberg_tables.
2025-01-28 16:20:03.657
java.io.IOException: Failure to create directory /datamesh/metadata.
2025-01-28 16:20:03.656
java.io.IOException: Failure to create directory /datamesh/node_history.
2025-01-28 16:18:05.387
java.io.IOException: Failure to create directory /datamesh/node_history.
2025-01-28 16:18:05.372
java.io.IOException: Failure to create directory /datamesh/accelerator.
2025-01-28 16:18:05.331
java.io.IOException: Failure to create directory /datamesh/data/results_cache.
2025-01-28 16:18:05.329
java.io.IOException: Failure to create directory /datamesh/scratch.
2025-01-28 16:18:05.327
java.io.IOException: Failure to create directory /datamesh/system_iceberg_tables.
2025-01-28 16:18:05.325
java.io.IOException: Failure to create directory /datamesh/metadata.
2025-01-28 16:18:05.325
java.io.IOException: Failure to create directory /datamesh/downloads.
2025-01-28 16:15:12.786
java.io.IOException: Failure to create directory /datamesh/accelerator.
2025-01-28 16:15:12.782
java.io.IOException: Failure to create directory /datamesh/system_iceberg_tables.
2025-01-28 16:15:12.781
java.io.IOException: Failure to create directory /datamesh/metadata.
2025-01-28 16:15:12.779
java.io.IOException: Failure to create directory /datamesh/node_history.
2025-01-28 16:15:12.778
java.io.IOException: Failure to create directory /datamesh/data/results_cache.
2025-01-28 16:15:12.774
java.io.IOException: Failure to create directory /datamesh/downloads.
2025-01-28 16:15:12.774
java.io.IOException: Failure to create directory /datamesh/scratch.