Dremio not starting - Lock file to RocksDB is currently hold

1

Hi - I am unable to start Dremio after an EC2 instance was stopped.

This is what I get when I run: dremio-admin clean -c

2018-06-18 11:58:01,845 [main] INFO c.d.datastore.LocalKVStoreProvider - Starting LocalKVStoreProvider
2018-06-18 11:58:02,093 [main] INFO c.dremio.datastore.ByteStoreManager - Lock file to RocksDB is currently hold by another process. Will wait until lock is freed.
Lock file to RocksDB is currently hold by another process. Will wait until lock is freed.

and: cat /var/log/dremio/server.out
Mon Jun 18 12:01:00 UTC 2018 Starting dremio on
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31105
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
/opt/dremio/bin/dremio: line 106: /var/run/dremio/dremio.pid: Permission denied
Lock file to RocksDB is currently hold by another process. Will wait until lock is freed.

Please can someone suggest a few checks to get dremio to run again.

Thanks in advance!

2

Can you verify that all Dremio processes are stopped? Seems like our internal metadata store is being held by another process. Btw you should only run dremio-admin clean when the service is stopped. You should also only run it as the (dremio) service account

3

Hi Anthony - the service is not running:
[centos@fra1uai01-a /]$ sudo service dremio status
dremio not running.

4

Futher info:

[centos@fra1uai01-a /] sudo service dremio stop** **no dremio to stop because no pid file /var/run/dremio/dremio.pid** **[centos@fra1uai01-a /]

5

Can you check the list of all running process and double check there’s no other instance of Dremio running? Would you happen to have another Dremio instance on another host somewhere?

1 Like
6

Hi - I am not sure if the java process is the cause:
[centos@fra1uai01-a /]$ ps -aux | less | grep dremio
dremio 5441 31.9 11.7 6195872 943140 ? Sl 07:01 0:22 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-8.b10.el7_5.x86_64/jre/bin/java -Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler -Djava.library.path=/opt/dremio/lib -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/dremio/server.gc -Ddremio.log.path=/var/log/dremio -Xmx4096m -XX:MaxDirectMemorySize=8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/dremio -cp /etc/dremio:/opt/dremio/jars/:/opt/dremio/jars/ext/:/opt/dremio/jars/3rdparty/ com.dremio.dac.daemon.DremioDaemon dremio start*
centos 6564 0.0 0.0 112704 968 pts/0 S+ 07:02 0:00 grep --color=auto dremio

There is only a single stand-alone Centos host for now.

7

Hi Anthony - I stopped all processes and ran the metadata cleanup. After that all started working agin. Many thanks for you help.

1 Like