Cleanup issue with lock

1

Hello,
I’m trying to do a cleanup of old jobs with the dremio-admin cleanup command but I get this error :

2018-02-23 15:38:58,613 [main] INFO c.d.datastore.LocalKVStoreProvider - Starting LocalKVStoreProvider
2018-02-23 15:38:58,758 [main] INFO c.d.datastore.LocalKVStoreProvider - Stopping LocalKVStoreProvider
2018-02-23 15:38:58,759 [main] INFO c.d.datastore.LocalKVStoreProvider - Stopped LocalKVStoreProvider
Failed to complete cleanup.
org.rocksdb.RocksDBException: lock /data/dremio/db/catalog/LOCK: Resource temporarily unavailable
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:286)
at com.dremio.datastore.ByteStoreManager.start(ByteStoreManager.java:144)
at com.dremio.datastore.CoreStoreProviderImpl.start(CoreStoreProviderImpl.java:167)
at com.dremio.datastore.LocalKVStoreProvider.start(LocalKVStoreProvider.java:86)
at com.dremio.dac.cmd.Clean.go(Clean.java:122)
at com.dremio.dac.cmd.Clean.main(Clean.java:167)

There’s a file named lock at this very specific location :
someone@somewhere:/data# ll /data/dremio/db/catalog/LOCK
-rw-r–r-- 1 dremio dremio 0 Jan 4 18:17 /data/dremio/db/catalog/LOCK

Does clean up requires stopping everything ?

2

Hi @ybi

Yes that is correct, Dremio needs to be down for this command to work

Thanks,
@balaji.ramaswamy

3

if Dremio should be down, how does it clean the executor nodes that are not on the same server ?
It appears “Cleanup should be run on master node”, when I try to run it on an executor node

4

Hi @dfleckinger

Yes, should be run on the master node, and master needs to be down

Metadata Cleanup

Thanks,
@balaji.ramaswamy

5

Thanks @balaji.ramaswamy.
But how are the executors nodes cleaned ? I’m havinn an executor node running out of space.

6

@dfleckinger,

Can you please send me output of df -h and also du -sh * of the folder that is occupying the most space

Thanks,
@balaji.ramaswamy

7

@balaji.ramaswamy, i’m sorry I had deleted files, so the command output has no sense anymore.
However i’m still curiuos to understand how an executor can be cleaned if it has to be offline during cleanup

8

The cleanup is only for Rocks DB KVstore on the master