Dremio trying to write results to wrong directory

1

Hello,

When I first installed Dremio I set it up for myself on my account on our cluster and created the dremio/data directory in my own home directory. Now I have moved dremio/data to a shared location on our scratch drive.

I have set up Dremio for other users to test out now and whenever they try to create a dataset it fails because it is still trying to write the results to the dremio/data/pdfs/results directory in my HOME directory and not the appropriate shared one on scratch.

I’ve added this line to my dremio.conf:

paths: {

the local path for dremio to store data.

local: “/scratch/dremio/dremio_data”
results: “pdfs://”${paths.local}"/results
}

I’ve looked through dremio-env as well and the path it is trying to write to is not defined anywhere. I’m guessing this has something to do with me copying the dremio/data directory from my home to scratch. Is there a way, without me reinstalling everything, to fix this problem (i.e. does anyone know where Dremio is being told to write results to $HOME/dremio/data/pdfs/results rather than scratch or how to overcome this problem)?

Any insight would be greatly appreciated!

Thanks in advance

2

@Ludah

I am assuming the one you pasted is the latest dremio.conf. In that you have results as below

results: “pdfs://”${paths.local}"/results

and also you have below

local: “/scratch/dremio/dremio_data”

So results should be written to

/scratch/dremio/dremio_data/results

Are you saying it is not? Have you restarted the coordinator after the change

Can you try hardcoding the path instead of using path.local just to narrow down the issue like below

results: “pdfs:///whatever-folder1/whatever-folder2/result” (you need 3 “///”)

3

@balaji.ramaswamy

Thank you for your response. I have made the change that you suggested and I still get the same error:

IOException: Mkdirs failed to create /home/mfa/dech/dremio_data/pdfs/results/2225cfe5-eba6-6d96-98ce-72c8363d2200

The paths in dremio.conf are set to:

paths: {
  local: "/scratch/dremio/dremio_data"
  results: "pdfs:///scratch/dremio/dremio_data/results"
}

The data set I am trying to open is a local directory. I should note that I copied over the original “dremio_data” directory from when I was doing testing and it was located in my $HOME directory (/home/mfa/dech/dremio_data …). My thinking was this was the cause of the issue. However, I’ve tried emptying the dremio_data directory and restarting Dremio but when I check the node that it should be running on the service is not started and the log shows the following error:

Dremio is exiting. Failure while starting services.
com.dremio.common.exceptions.UserException: Tried to access non-existent source       [__jobResultsStore].

Thanks again.

4

@Ludah

Try this, remove the data folder, then explicitly call out all components like below and restart Dremio

paths: {

the local path for dremio to store data.

local: “/Users/ludah/dremio_metadata/data”

the distributed path Dremio data including job results, downloads, uploads, etc

accelerator: “pdfs:///Users/ludah/dremio_pdfs//accelerator”
uploads: “pdfs:///Users/ludah/dremio_pdfs//uploads”
downloads: “pdfs:///Users/ludah/dremio_pdfs/downloads”
scratch: “pdfs:///Users/ludah/dremio_pdfs/scratch”
results: “pdfs:///Users/ludah/dremio_pdfs/results”
}

You can use any path you want but lets explicitly set all of the above and local

5

@balaji.ramaswamy

Thank you very much! Everything seems to be working now.