Can't stop node

1

Hi,

I have Dremio running in Docker from the image: dremio/dremio-oss:latest, build: Build 24.3.0-202312220021110714-f19d865e.
I want to execute the command dremio-admin clean -d=7, but I’m getting an error: Lock file to RocksDB is currently held by another process. Stop other process before retrying.

The dremio status command indicates that the service is stopped, but the UI is working and I can perform all actions.

I’ve checked the list of processes, and there’s a process created by Dremio with PID 1

dremio pid
dremio pid1574×56 4.25 KB

that I can’t stop in any way.

I tested the ‘dremio start’ command, which starts a new node. Then, ‘dremio status’ shows ‘working’. Stopping the node with ‘dremio stop’ does not affect the process running under PID 1 in any way.

What can I do to execute the clean command?

2

@przemyslaw_szymanski Couple of things here

  • For unable to stop Dremio, I am wondering if an orphan process is running, can you so a ps -ef | grep dremio and send us the outpiut

  • What is the reason you are doing dremio-admin clean -d 7?

Thanks
Bali

3

image
image1572×255 11.6 KB

I want to delete data from old queries/reflections that are taking up a lot of disk space, and at the moment, I don’t need them.

4

@przemyslaw_szymanski Do you see the above output after shutting down Dremio? If that is true then do a kill -9 1

For the clean, let us take one by one

queries - I assume you mean jobs and profiles, let us first see how much space jobs/profiles are taking. Do you have support optiion planner.verbose_profile enabled? The -d does not clean profiles/jobs

Would you be able to send me the zip attachment created by this command, it tells us what is occupying space in your KVstore, change token

curl --location --request GET 'localhost:9047/apiv2/kvstore/report?store=none' \
--header 'Authorization: _dremioirr8hj6qnpc3tfr3omiqvev51c' > kvstore_summary.zip

Reflections, the actual parquet files are not in the KVstore and clean -d again will not help. The only thing for reflections inside the KVstore are the reflection definitions and should not use much space and the above report will tell us

Also would you be able to share your dremio.conf that will tell us where your reflections are? You can X out any passwords

Lastly, send the output of the below 2 queries, download as PARQUET

SELECT * FROM sys.materializations
SELECT * FROM sys.reflections

Thanks,
Bali