Dremio server crashes after 3 seconds

Hello Dremio Community,

currently I am setting up Dremio on an AWS machine. I was able to start the service and status looks good immediately after launching it:
● dremio.service - Dremio Daemon Server
Loaded: loaded (/etc/systemd/system/dremio.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2018-06-11 09:26:52 UTC; 2s ago
Docs: https://docs.dremio.com
Process: 3385 ExecStop=/opt/dremio/bin/dremio stop (code=exited, status=0/SUCCESS)
Main PID: 3668 (java)
Tasks: 10
Memory: 80.0M
CPU: 2.091s
CGroup: /system.slice/dremio.service
└─3668 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler -Djava.library.path=/opt/dremio/lib -XX:+Pri

Jun 11 09:26:52 ip-172-30-0-244 systemd[1]: Started Dremio Daemon Server.
Jun 11 09:26:52 ip-172-30-0-244 dremio[3668]: starting dremio, logging to var/log/dremio/server.out


However, it crashes within 3 seconds. I get this status

● dremio.service - Dremio Daemon Server
Loaded: loaded (/etc/systemd/system/dremio.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2018-06-11 09:26:58 UTC; 15s ago
Docs: https://docs.dremio.com
Process: 3751 ExecStop=/opt/dremio/bin/dremio stop (code=exited, status=0/SUCCESS)
Process: 3668 ExecStart=/opt/dremio/bin/dremio start-fg (code=exited, status=1/FAILURE)
Main PID: 3668 (code=exited, status=1/FAILURE)

Jun 11 09:26:52 ip-172-30-0-244 systemd[1]: Started Dremio Daemon Server.
Jun 11 09:26:52 ip-172-30-0-244 dremio[3668]: starting dremio, logging to var/log/dremio/server.out
Jun 11 09:26:58 ip-172-30-0-244 systemd[1]: dremio.service: Main process exited, code=exited, status=1/FAILURE
Jun 11 09:26:58 ip-172-30-0-244 dremio[3751]: no dremio to stop because kill -0 of pid 3668 failed with status 1
Jun 11 09:26:58 ip-172-30-0-244 systemd[1]: dremio.service: Unit entered failed state.
Jun 11 09:26:58 ip-172-30-0-244 systemd[1]: dremio.service: Failed with result ‘exit-code’.


Looking at the most recent entry in server.out it seems like a LOCK file can’t be opened:

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 3900
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 3900
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Exception in thread “main” org.rocksdb.RocksDBException: while open a file for lock: /opt/dremio/data/db/catalog/LOCK: Permission denied
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:286)
at com.dremio.datastore.ByteStoreManager.openDB(ByteStoreManager.java:204)
at com.dremio.datastore.ByteStoreManager.start(ByteStoreManager.java:160)
at com.dremio.datastore.CoreStoreProviderImpl.start(CoreStoreProviderImpl.java:167)
at com.dremio.datastore.LocalKVStoreProvider.start(LocalKVStoreProvider.java:88)
at com.dremio.dac.daemon.DremioDaemon.checkVersion(DremioDaemon.java:86)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:125)


The most recent entry in the according dremio.log file is:

2018-06-11 08:45:27,965 [main] INFO c.d.common.scanner.BuildTimeScan - Loaded prescanned packages [com.dremio.service.jobs, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.service.namespace, com.dremio.service.users, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.dac, com.dremio.dac.support.SupportService, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.provision, com.dremio.service.accelerator, com.dremio.service.reflection, com.dremio.service.voting, com.dremio.service.jobs, com.dremio.service.namespace, com.dremio.service.users, com.dremio.exec.store, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.storage, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.storage, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.service.namespace, com.dremio.service.users, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.storage, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.storage, org.apache.hadoop.hive, com.dremio.exec.fn.hive, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.service.namespace, com.dremio.service.users, org.apache.hadoop.hive, com.dremio.sabot.task.slicing.SlicingTaskPool, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.service.namespace, com.dremio.service.users, com.dremio.service.accelerator, com.dremio.service.reflection, com.dremio.service.voting, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.service.jobs, com.dremio.service.namespace, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.service.users, com.dremio.exec.store] from locations [jar:file:/opt/dremio/jars/dremio-services-jobs-2.0.5-201806021755080191-767cfb5.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-dac-backend-2.0.5-201806021755080191-767cfb5.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-services-datastore-2.0.5-201806021755080191-767cfb5.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-dac-common-2.0.5-201806021755080191-767cfb5.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-common-2.0.5-201806021755080191-767cfb5.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-sabot-logical-2.0.5-201806021755080191-767cfb5.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-hive-plugin-2.0.5-201806021755080191-767cfb5.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-extra-sabot-scheduler-2.0.5-201806021755080191-767cfb5.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-services-accelerator-2.0.5-201806021755080191-767cfb5.jar!/META-INF/dremio-module-scan/registry.json]
2018-06-11 08:45:29,469 [main] INFO c.d.common.scanner.ClassPathScanner - Scanning packages [com.dremio.plugins.s3.store, com.dremio.plugins.adl, com.dremio.service.jobs, com.dremio.service.users, com.dremio.exec.store.jdbc, com.dremio.dac, com.dremio.dac.support.SupportService, com.dremio.plugins.mongo, com.dremio.provision.yarn.service, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.plugins.elastic, com.dremio.exec.store, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.extras.plugins.elastic, org.apache.hadoop.hive, com.dremio.exec.fn.hive, com.dremio.exec.store.hbase, com.dremio.exec.expr.fn.impl.conv, com.dremio.service.namespace, com.dremio.provision, com.dremio.plugins.mongo, com.dremio.exec.store.jdbc, com.dremio.sabot.task.slicing.SlicingTaskPool, com.dremio.service.accelerator, com.dremio.service.reflection, com.dremio.service.voting, com.dremio.extra.exec.store.dfs, com.dremio.exec.planner.acceleration.substitution, org.apache.hadoop.hive] in locations [jar:file:/opt/dremio/jars/dremio-s3-plugin-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/dremio-services-users-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/dremio-services-coordinator-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/dremio-extra-plugin-jdbc-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/dremio-extra-plugin-mongo-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/dremio-yarn-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/dremio-elasticsearch-plugin-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/dremio-pdfs-plugin-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/dremio-extra-plugin-elasticsearch-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/dremio-hbase-plugin-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/dremio-services-namespace-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/dremio-provision-common-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/dremio-mongo-plugin-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/dremio-jdbc-plugin-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/dremio-extra-plugin-hive-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/dremio-extra-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar!/, jar:file:/opt/dremio/jars/3rdparty/dremio-hive-exec-shaded-2.0.5-201806021755080191-767cfb5.jar!/] took 1491ms
2018-06-11 08:45:29,544 [main] INFO c.d.datastore.LocalKVStoreProvider - Starting LocalKVStoreProvider
2018-06-11 08:45:29,728 [main] INFO c.d.datastore.LocalKVStoreProvider - Stopping LocalKVStoreProvider
2018-06-11 08:45:29,731 [main] INFO c.d.datastore.LocalKVStoreProvider - Stopped LocalKVStoreProvider


Also I can provide the access permissions for the LOCK file @ /opt/dremio/data/db/catalog:
-rwxr-xr-x 1 hadoop_user hadoop_group 0 Jun 8 14:44 LOCK

Any help is appreciated!

Thanks in advance!

Reinstalled using this webpage

then it worked.

Hi @isSokOl

Let’s do this,

shutdown Dremio
ps -ef | grep dremio

Make sure nothing is running

./start dremio
tail -f server.out

Send us the server.out and server.log

Thanks
@balaji.ramaswamy

/opt/dremio/bin/dremio: line 106: /var/run/dremio/dremio.pid: No such file or directory
RocksDBExceptionJni::ThrowNew/StatusJni - Error: unexpected exception!
Exception in thread “main” java.lang.IllegalArgumentException: Illegal value provided for SubCode.
at org.rocksdb.Status$SubCode.getSubCode(Status.java:109)
at org.rocksdb.Status.(Status.java:30)
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:286)
at com.dremio.datastore.ByteStoreManager.openDB(ByteStoreManager.java:173)
at com.dremio.datastore.ByteStoreManager.start(ByteStoreManager.java:148)
at com.dremio.datastore.CoreStoreProviderImpl.start(CoreStoreProviderImpl.java:167)
at com.dremio.datastore.LocalKVStoreProvider.start(LocalKVStoreProvider.java:88)
at com.dremio.dac.daemon.DremioDaemon.checkVersion(DremioDaemon.java:86)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:125)
Sun Aug 12 16:09:35 IDT 2018 Starting dremio on ubuntu
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 2015465
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 2015465
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
RocksDBExceptionJni::ThrowNew/StatusJni - Error: unexpected exception!
Exception in thread “main” java.lang.IllegalArgumentException: Illegal value provided for SubCode.
at org.rocksdb.Status$SubCode.getSubCode(Status.java:109)
at org.rocksdb.Status.(Status.java:30)
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:286)
at com.dremio.datastore.ByteStoreManager.openDB(ByteStoreManager.java:173)
at com.dremio.datastore.ByteStoreManager.start(ByteStoreManager.java:148)
at com.dremio.datastore.CoreStoreProviderImpl.start(CoreStoreProviderImpl.java:167)
at com.dremio.datastore.LocalKVStoreProvider.start(LocalKVStoreProvider.java:88)
at com.dremio.dac.daemon.DremioDaemon.checkVersion(DremioDaemon.java:86)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:125)

Any idea how to clean rocks db state

Hi @shachar

Lets check a few things

  1. Make sure, no dremio process or orphaned pid’s are running

ps -ef | grep dremio

If it is running and shutting down Dremio does not work, then just kill the process

  1. Make sure /var/run/dremio exists with the right permissions? Process owning Dremio should be able to write into that folder

  2. start dremio and tail the server.out and see if Dremio starts or send us the server.out and server.log

Thanks
@balaji.ramaswamy

Thanks

I shutdown the system

cleaned all data files

And restarted the server

It wasn’t painful since it is a test data

Best

Shachar