Dremio CE 22.1 & distributed storage

1

Hello everyone,
I’ve a kubernetes cluster with dremio community (version 22.1).
I’ve enabled distributed storage with s3 as backend but after this some queries stuck without completing.

Some advice?

2

@eperrella How many executors do you have? Are you able to send the job profile of the stuck job?

3

@balaji.ramaswamy thanks for your time. Actually we have 2 executors (it’s a develop configuration).
I try to download & upload it.

4

@balaji.ramaswamy an example of profile
44decea8-8940-49f7-8e1e-8d38dab65923.zip (29.9 KB)

5

@eperrella It looks like these executors are not responding, can we get the server.log and all GC files on these executors and see if there are any errors or there is a full GC going on?

6

thanks for your response. Of course we can do it, but this kind of problems are coming after the switch of the storage from local to distributed.

7

I’ve found only this line in one of the executors:

Unable to write cache chunk to disk due to storage-plugin not found.

do you know somethings about this?

8

@eperrella Are you able to strip out sensitive information from dremio.conf?

9

Of course, here in attachment our dremio’s configuration file.

dremio.zip (1.1 KB)

10

@eperrella Where is the writing to? Does Dremio have permission? Can you send ls -ltrh dist: ${STORAGE_BACKEND}"dremio-storage-"${ENVIRONMENT}/"results"?