Dremio not running

1

I cannot start dremio,

dremio status shows:
/opt/dremio/run/dremio.pid file is present but dremio not running.

server.out:
Mon Oct 29 14:09:56 EDT 2018 Starting dremio on ulv100137.devfg.rbc.com
core file size (blocks, -c) 10485760
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 127892
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 10240
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
KVStore version is 2.1.6-201809161906440178-edb5b4d
Catastrophic failure occurred. Exiting. Information follows: Failed to start services, daemon exiting.
com.dremio.common.exceptions.UserException: The source ["__jobResultsStore"] is currently unavailable. Info: [[Message{level=ERROR, msg=Failure to create directory /dremio/pdfs/results.}]].
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746)
at com.dremio.exec.catalog.ManagedStoragePlugin.checkState(ManagedStoragePlugin.java:340)
at com.dremio.exec.catalog.ManagedStoragePlugin.unwrap(ManagedStoragePlugin.java:579)
at com.dremio.exec.catalog.CatalogServiceImpl.getSource(CatalogServiceImpl.java:847)
at com.dremio.dac.daemon.DACDaemonModule$4.get(DACDaemonModule.java:385)
at com.dremio.dac.daemon.DACDaemonModule$4.get(DACDaemonModule.java:380)
at com.dremio.service.jobs.LocalJobsService.start(LocalJobsService.java:259)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:177)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:183)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:112)

server.log:
2018-10-29 14:10:41,396 [main] INFO c.d.service.jobs.LocalJobsService - Starting JobsService
2018-10-29 14:10:41,415 [main] INFO c.d.e.catalog.ManagedStoragePlugin - User Error Occurred [ErrorId: 621ad867-2877-44fe-9fdc-07eecc4a204a]
com.dremio.common.exceptions.UserException: The source ["__jobResultsStore"] is currently unavailable. Info: [[Message{level=ERROR, msg=Failure to create directory /dremio/pdfs/results.}]].
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.exec.catalog.ManagedStoragePlugin.checkState(ManagedStoragePlugin.java:340) [dremio-sabot-kernel-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.exec.catalog.ManagedStoragePlugin.unwrap(ManagedStoragePlugin.java:579) [dremio-sabot-kernel-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.exec.catalog.CatalogServiceImpl.getSource(CatalogServiceImpl.java:847) [dremio-sabot-kernel-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.dac.daemon.DACDaemonModule$4.get(DACDaemonModule.java:385) [dremio-dac-backend-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.dac.daemon.DACDaemonModule$4.get(DACDaemonModule.java:380) [dremio-dac-backend-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.service.jobs.LocalJobsService.start(LocalJobsService.java:259) [dremio-services-jobs-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137) [dremio-common-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74) [dremio-common-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33) [dremio-common-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:177) [dremio-dac-backend-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:183) [dremio-dac-backend-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:112) [dremio-dac-daemon-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
2018-10-29 14:10:41,419 [main] ERROR c.dremio.common.CatastrophicFailure - Catastrophic Failure Occurred, exiting. Information message: Failed to start services, daemon exiting.
com.dremio.common.exceptions.UserException: The source ["__jobResultsStore"] is currently unavailable. Info: [[Message{level=ERROR, msg=Failure to create directory /dremio/pdfs/results.}]].
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.exec.catalog.ManagedStoragePlugin.checkState(ManagedStoragePlugin.java:340) ~[dremio-sabot-kernel-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.exec.catalog.ManagedStoragePlugin.unwrap(ManagedStoragePlugin.java:579) ~[dremio-sabot-kernel-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.exec.catalog.CatalogServiceImpl.getSource(CatalogServiceImpl.java:847) ~[dremio-sabot-kernel-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.dac.daemon.DACDaemonModule$4.get(DACDaemonModule.java:385) ~[dremio-dac-backend-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.dac.daemon.DACDaemonModule$4.get(DACDaemonModule.java:380) ~[dremio-dac-backend-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.service.jobs.LocalJobsService.start(LocalJobsService.java:259) ~[dremio-services-jobs-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137) ~[dremio-common-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74) ~[dremio-common-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33) ~[dremio-common-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:177) ~[dremio-dac-backend-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:183) ~[dremio-dac-backend-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:112) ~[dremio-dac-daemon-2.1.6-201809161906440178-edb5b4d.jar:2.1.6-201809161906440178-edb5b4d]

2

Hi @carol

It may be that the user starting the Dremio daemon does not have read/write permissions to the directories in the error message:

Info: [[Message{level=ERROR, msg=Failure to create directory /dremio/pdfs/results.}]].

Check that the Dremio user has those permissions.

3

Hello @ben
I change the path to store the results in hdfs in dremio.config, so I assume that I dont need /dremio/pdfs/results.
But i dont know why it is not able to write the results in a hdfs.

Thanks

4

@carol Can you share your settings from “dremio.conf”? It looks like Dremio is defaulting to standard values for those paths so perhaps something is incorrect in the configuration syntax.

5

@ben
Hello I figured it out. The problem was my hdfs node was down. Now it works

Thank you