Unable to start Dremio: java.lang.NullPointerException: No Cluster Identity found

I installed dremio-community-1.4.4-201801230630390665_6d69d32_mapr_1.noarch.rpm. i was able to connect to a secure Mapr cluster. I created yarn provisioning, executors were running on resource manager of MapR. Things were running fine till i did kill -9 <pid>. Now dremio never starts. This is centos 7 operating system.

i deleted the below folders

rm -rf /var/lib/dremio/*
rm -rf /var/log/dremio/*
rm -rf /var/run/dremio/*

however, the error persists

Tue Feb  6 11:03:54 EST 2018 Starting dremio on mtl-prep-dremio-01.preprod.stp.local
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127969
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
Exception in thread "main" java.lang.NullPointerException: No Cluster Identity found
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:226)
        at com.dremio.dac.daemon.DremioDaemon.checkVersion(DremioDaemon.java:106)
        at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:142)

There are no errors listed in server.log. There is a valid mapr ticket created on the dremio box. I am not sure what i am missing. Can you kindly help me out of this issue?

Do you mind sharing your dremio.conf configuration file (under /etc/dremio)?

here it is

paths: {
local: “/var/lib/dremio”

}

services: {
coordinator.enabled: true,
coordinator.auto-upgrade=false,
coordinator.client-endpoint.port: 31020,
coordinator.master.enabled: true,
executor.enabled: false
}

services.coordinator.master.embedded-zookeeper.enabled: false

zookeeper: “mtl-prep-mapr-01.preprod.stp.local:5181,mtl-prep-mapr-02.preprod.stp.local:5181,mtl-prep-mapr-03.preprod.stp.local:5181,mtl-prep-mapr-04.preprod.stp.local:5181”

At times i see the below erro. Just incase it helps in identifying the issue

2018-02-06 12:02:18,822 [main] WARN  com.dremio.datastore.StoreLoader - Unable to load StoreCreationFunction JobsProfileCreator
com.dremio.datastore.DatastoreException: java.util.concurrent.ExecutionException: java.io.IOException: failure to login: No LoginModules configured for hadoop_default
        at com.dremio.datastore.CoreStoreProviderImpl.checkedGet(CoreStoreProviderImpl.java:286) ~[dremio-services-datastore-1.4.4-201801230630390665-6d69d32-mapr.jar:1.4.4-201801230630390665-6d69d32-mapr]
        at com.dremio.datastore.CoreStoreProviderImpl.access$800(CoreStoreProviderImpl.java:60) ~[dremio-services-datastore-1.4.4-201801230630390665-6d69d32-mapr.jar:1.4.4-201801230630390665-6d69d32-mapr]
        at com.dremio.datastore.CoreStoreProviderImpl$CoreStoreBuilderImpl.build(CoreStoreProviderImpl.java:253) ~[dremio-services-datastore-1.4.4-201801230630390665-6d69d32-mapr.jar:1.4.4-201801230630390665-6d69d32-mapr]
        at com.dremio.datastore.LocalKVStoreProvider$LocalStoreBuilder.build(LocalKVStoreProvider.java:187) ~[dremio-services-datastore-1.4.4-201801230630390665-6d69d32-mapr.jar:1.4.4-201801230630390665-6d69d32-mapr]
        at com.dremio.service.jobs.LocalJobsService$JobsProfileCreator.build(LocalJobsService.java:1497) ~[dremio-services-jobs-1.4.4-201801230630390665-6d69d32-mapr.jar:1.4.4-201801230630390665-6d69d32-mapr]
        at com.dremio.datastore.StoreLoader.buildStores(StoreLoader.java:43) ~[dremio-services-datastore-1.4.4-201801230630390665-6d69d32-mapr.jar:1.4.4-201801230630390665-6d69d32-mapr]
        at com.dremio.datastore.LocalKVStoreProvider.start(LocalKVStoreProvider.java:99) [dremio-services-datastore-1.4.4-201801230630390665-6d69d32-mapr.jar:1.4.4-201801230630390665-6d69d32-mapr]
        at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137) [dremio-common-1.4.4-201801230630390665-6d69d32-mapr.jar:1.4.4-201801230630390665-6d69d32-mapr]
        at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74) [dremio-common-1.4.4-201801230630390665-6d69d32-mapr.jar:1.4.4-201801230630390665-6d69d32-mapr]
        at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33) [dremio-common-1.4.4-201801230630390665-6d69d32-mapr.jar:1.4.4-201801230630390665-6d69d32-mapr]
        at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:174) [dremio-dac-backend-1.4.4-201801230630390665-6d69d32-mapr.jar:1.4.4-201801230630390665-6d69d32-mapr]
        at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:180) [dremio-dac-backend-1.4.4-201801230630390665-6d69d32-mapr.jar:1.4.4-201801230630390665-6d69d32-mapr]
        at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:164) [dremio-dac-daemon-1.4.4-201801230630390665-6d69d32-mapr.jar:1.4.4-201801230630390665-6d69d32-mapr]
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: failure to login: No LoginModules configured for hadoop_default
        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2320) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
        at com.dremio.datastore.CoreStoreProviderImpl.checkedGet(CoreStoreProviderImpl.java:283) ~[dremio-services-datastore-1.4.4-201801230630390665-6d69d32-mapr.jar:1.4.4-201801230630390665-6d69d32-mapr]
        ... 12 common frames omitted
Caused by: java.io.IOException: failure to login: No LoginModules configured for hadoop_default
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:751) ~[hadoop-common-2.7.0-mapr-1703.jar:na]
        at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:688) ~[hadoop-common-2.7.0-mapr-1703.jar:na]
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:572) ~[hadoop-common-2.7.0-mapr-1703.jar:na]

Could you also share dremio-env under /etc/dremio?
No LoginModules configured somehow point to the fact that either mapr.login.conf is not on the path or it can’t get it from jar. Also looks like your cluster is with security enabled, so it’s definitely need non-default mapr.login.conf on the classpath

export MAPR_IMPERSONATION_ENABLED=true
export DREMIO_HOME=/opt/dremio/
export MAPR_TICKETFILE_LOCATION=/tmp/maprticket_5000
export DREMIO_JAVA_EXTRA_OPTS=" -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf "

Please update it with highlighted areas and try again

1 Like

Wow! the server is up!

But how did it work previously? and why did allstop working when i did kill -9

hey!

but after restart, say i configure maprfs, followed by yarn and then restart the dremio… it doesnt comeup again

Tue Feb  6 19:19:00 EST 2018 Starting dremio on mtl-prep-dremio-01.preprod.stp.local
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127969
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
Exception in thread "main" java.lang.NullPointerException: No Cluster Identity found
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:226)
        at com.dremio.dac.daemon.DremioDaemon.checkVersion(DremioDaemon.java:106)
        at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:142)

vim /etc/dremio/dremio-env

if [ -f /opt/mapr/conf/env.sh ]; then
. /opt/mapr/conf/env.sh
fi

export MAPR_IMPERSONATION_ENABLED=true
export DREMIO_HOME=/opt/dremio/
export MAPR_TICKETFILE_LOCATION=/tmp/maprticket_5000
#export DREMIO_JAVA_EXTRA_OPTS=" -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf "

export DREMIO_CLASSPATH_USER_FIRST=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop
DREMIO_JAVA_EXTRA_OPTS="${DREMIO_JAVA_EXTRA_OPTS} ${MAPR_JAAS_CONFIG_OPTS} ${MAPR_ZOOKEEPER_OPTS}"

I doubt “kill -9” had anything to do with misconfiguration. Those two are unrelated. I was giving you advice based on “No LoginModules” - that was clearly MapR misconfiguration. Regarding “No Cluster Identity found”. Can’t say much. What server.log shows?

not much…

2018-02-06 19:17:41,940 [main] INFO  c.d.datastore.LocalKVStoreProvider - Starting LocalKVStoreProvider
2018-02-06 19:17:44,115 [main] INFO  c.d.datastore.LocalKVStoreProvider - LocalKVStoreProvider is up
2018-02-06 19:17:44,116 [main] INFO  c.d.datastore.LocalKVStoreProvider - Stopping LocalKVStoreProvider
2018-02-06 19:17:44,136 [main] INFO  c.d.datastore.LocalKVStoreProvider - Stopped LocalKVStoreProvider
2018-02-06 19:19:01,758 [main] INFO  com.dremio.exec.util.GuavaPatcher - Google's Stopwatch patched for old HBase Guava version.
2018-02-06 19:19:01,778 [main] INFO  com.dremio.exec.util.GuavaPatcher - Google's Closeables patched for old HBase Guava version.
2018-02-06 19:19:01,863 [main] INFO  com.dremio.common.config.SabotConfig - Configuration and plugin file(s) identified in 78ms.
Base Configuration:
        - jar:file:/opt/dremio/jars/dremio-common-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-default.conf

Intermediate Configuration and Plugin files, in order of precedence:
        - jar:file:/opt/dremio/jars/dremio-jdbc-plugin-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-services-users-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-yarn-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-sabot-logical-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-hive-plugin-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-s3-plugin-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-pdfs-plugin-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-elasticsearch-plugin-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-services-datastore-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-services-coordinator-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-extra-plugin-jdbc-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-dac-backend-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-services-namespace-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-services-accelerator-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-common-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-dac-common-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-services-jobs-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-sabot-kernel-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-extra-plugin-mongo-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-mongo-plugin-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-provision-common-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-extra-sabot-scheduler-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-hbase-plugin-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-extra-plugin-hive-1.4.4-201801230630390665-6d69d32-mapr.jar!/sabot-module.conf


2018-02-06 19:19:02,104 [main] INFO  c.d.common.scanner.BuildTimeScan - Loaded prescanned packages [com.dremio.service.users, com.dremio.sabot.task.slicing.SlicingTaskPool, com.dremio.exec.
ExecConstants, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.sto
re, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.service.namespace, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.storage, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.service.jobs, com.dremio.service.accelerator, com.dremio.service.users, com.dremio.exec.ExecConstants, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.exec.store, com.dremio.service.namespace, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.service.jobs, com.dremio.service.users, com.dremio.dac, com.dremio.dac.support.SupportService, com.dremio.exec.ExecConstants, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.provision, com.dremio.service.namespace, com.dremio.service.accelerator, com.dremio.exec.store, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.storage, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.service.users, com.dremio.storage, com.dremio.exec.ExecConstants, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.service.namespace, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.service.jobs, com.dremio.service.users, com.dremio.exec.ExecConstants, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.service.namespace, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs] from locations [jar:file:/opt/dremio/jars/dremio-extra-sabot-scheduler-1.4.4-201801230630390665-6d69d32-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-sabot-logical-1.4.4-201801230630390665-6d69d32-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-services-accelerator-1.4.4-201801230630390665-6d69d32-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-dac-backend-1.4.4-201801230630390665-6d69d32-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-dac-common-1.4.4-201801230630390665-6d69d32-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-common-1.4.4-201801230630390665-6d69d32-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-services-datastore-1.4.4-201801230630390665-6d69d32-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-sabot-kernel-1.4.4-201801230630390665-6d69d32-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-services-jobs-1.4.4-201801230630390665-6d69d32-mapr.jar!/META-INF/dremio-module-scan/registry.json]

2018-02-06 19:19:02,797 [main] INFO  c.d.common.scanner.ClassPathScanner - Scanning packages [com.dremio.exec.store.jdbc, com.dremio.service.users, com.dremio.provision.yarn.service, org.apache.hadoop.hive, com.dremio.exec.fn.hive, com.dremio.plugins.s3.store, com.dremio.exec.store, com.dremio.plugins.elastic, com.dremio.exec.store.jdbc, com.dremio.dac, com.dremio.dac.support.SupportService, com.dremio.service.namespace, com.dremio.service.accelerator, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.service.jobs, com.dremio.exec.ExecConstants, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.plugins.mongo, com.dremio.plugins.mongo, com.dremio.provision, com.dremio.sabot.task.slicing.SlicingTaskPool, com.dremio.exec.store.hbase, com.dremio.exec.expr.fn.impl.conv, org.apache.hadoop.hive, com.dremio.extra.exec.store.dfs, com.dremio.exec.planner.acceleration.substitution, com.dremio.extras.plugins.elastic] in locations [jar:file:/opt/dremio/jars/dremio-mongo-plugin-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-extra-plugin-mongo-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-hive-plugin-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-yarn-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-services-namespace-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-services-coordinator-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-elasticsearch-plugin-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-extra-plugin-hive-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/3rdparty/dremio-hive-exec-shaded-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-extra-plugin-elasticsearch-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-pdfs-plugin-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-s3-plugin-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-extra-plugin-jdbc-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-jdbc-plugin-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-hbase-plugin-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-extra-sabot-kernel-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-provision-common-1.4.4-201801230630390665-6d69d32-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-services-users-1.4.4-201801230630390665-6d69d32-mapr.jar!/] took 690ms
2018-02-06 19:19:02,830 [main] INFO  c.d.datastore.LocalKVStoreProvider - Starting LocalKVStoreProvider
2018-02-06 19:19:05,104 [main] INFO  c.d.datastore.LocalKVStoreProvider - LocalKVStoreProvider is up
2018-02-06 19:19:05,106 [main] INFO  c.d.datastore.LocalKVStoreProvider - Stopping LocalKVStoreProvider
2018-02-06 19:19:05,132 [main] INFO  c.d.datastore.LocalKVStoreProvider - Stopped LocalKVStoreProvider

Are permissions on this directory OK for your user 5000 ?
Also I would recommend for MapR do a bit different configuration:
Dremio (you may not want to do YARN deployment if you don’t want to, but other config settings might come handy)

1 Like

Yes the permissions are ok. in dremio.conf I had changed the local variable (below). Apparently it already had a db in it. Removed it(db folder) and restarted; things are fine. I am not sure if i re-restart again things will be fine or not. Also, why I am changing this variable is that… on my mapr cluster during the yarn job, on mapr nodes, it seems like the job is spilling to /var/lib/dremio (local folder) which are not present on the cluster.

btw, I want dremio to use the mapr cluster. I dont want to create another cluster for this activity. So yarn is the only way to go for me. my user id is 5000 nd it has access to /var/lib/dremio folder

paths: {
  # the local path for dremio to store data.
  #local: "/var/lib/dremio"
  local:"/tmp/dremio"

I did not quite understand if you are using YARN for dremio right now or not?
If you do then doc page I gave a link earlier should have a screenshot of provisioning screen that will fill out spill directory based on mapr local volumes

If you do not use YARN you will need to define spill directory on each node. It is advisable to use mapr local volume location for it:
By default it is:
spilling: [${paths.local}/spill]

See: https://docs.dremio.com/advanced-administration/configuration-files.html

yes i am using Yarn for Dremio right now. Yes the screen shot in the link resembles the one i have. Then may be I am wrong with the spill directory issue. Now i dont have the logs to recheck. I will post the issue if it recurs again. Sorry for the noise