Hi,
We put working a POC dremio cluster using s3 as dist path to accelerator, results and etc. However, the s3 bucket we are using is used for other applications too and we need to separate dremio to use it own bucket.
The problem is when we change the conf file and apparently nothing happens. Dremio still saving reflections and query results into old bucket. I’ve try remove access permission of ec2 policy that dremio runs for that old bucket but dremio crashs on startup.
I need to known if dremio have some cache of this config and how to cleanup it. I’ve try all dremio-admin clean metadata options with no success. I’ve tried dremio-admin upgrade KVStore too, with no success.
My dremio version is 3.1.0. We are planning update version, but we’ll do it on EKS using dremio-cloud-tools. Anyway we need this solved because we’ll migrate the metadata using backups (or we intend).
My paths conf:
paths: {
# the local path for dremio to store data.
local: "/data/dremio"
spilling: ["/data/dremio/spill"],
# the distributed path Dremio data including job results, downloads, uploads, etc
dist: "s3://<new_bucket>/reflections",
accelerator: ${paths.dist}/accelerator,
downloads: ${paths.dist}/downloads,
uploads: ${paths.dist}/uploads,
results: ${paths.dist}/results
# location for catalog database (if master node)
db: ${paths.local}/db
}
The error when I removed the old bucket permission:
com.dremio.common.exceptions.UserException: The source ["__jobResultsStore"] is currently unavailable. Info: [[Message{level=ERROR, msg=Failure to create directory /dremio/results.}]].
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746)
at com.dremio.exec.catalog.ManagedStoragePlugin.checkState(ManagedStoragePlugin.java:347)
at com.dremio.exec.catalog.ManagedStoragePlugin.unwrap(ManagedStoragePlugin.java:588)
at com.dremio.exec.catalog.CatalogServiceImpl.getSource(CatalogServiceImpl.java:867)
at com.dremio.dac.daemon.DACDaemonModule$3.get(DACDaemonModule.java:407)
at com.dremio.dac.daemon.DACDaemonModule$3.get(DACDaemonModule.java:402)
at com.dremio.service.jobs.LocalJobsService.start(LocalJobsService.java:273)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102)
Thanks;