Unable to start Dremio in UNIX - KVStore has an older version

1

Hi,
Not able to start Dremio in UNIX server. Can you please help on below error.

PFB server.out logs:

Thu Sep 27 13:28:33 EDT 2018 Starting dremio on cspinf-wc-1p
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63678
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 63678
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
KVStore version is 1.4.9-201802191836310213-7195059
Catastrophic failure occurred. Exiting. Information follows: Failed to start services, daemon exiting.
java.lang.IllegalStateException: KVStore has an older version (1.4.9-201802191836310213-7195059) than the server (2.1.6-201809161911340177-edb5b4d-mapr), please run the upgrade tool first
at com.google.common.base.Preconditions.checkState(Preconditions.java:721)
at com.dremio.dac.daemon.DremioDaemon$AutoUpgrade.ensureUpgradeSupported(DremioDaemon.java:72)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:182)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:171)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:103)

Any help greatly appreciated.

Thanks,
hareesh

2

Upgrade tool is .../bin/dremio-admin upgrade

3

Thanks @yufeldman i executed upgrade tool. But while starting dremio getting below error. Can you please help on this.

file locks (-x) unlimited
KVStore version is 2.1.6-201809161911340177-edb5b4d-mapr
Catastrophic failure occurred. Exiting. Information follows: Failed to start services, daemon exiting.
java.lang.RuntimeException: java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: MapR user ticket not available! error = com.mapr.security.MutableInt@19cd7577
at com.google.common.base.Throwables.propagate(Throwables.java:240)
at com.dremio.dac.daemon.ZkServer.init(ZkServer.java:111)
at com.dremio.dac.daemon.ZkServer.start(ZkServer.java:74)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startPreServices(DACDaemon.java:172)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:182)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:112)
Caused by: java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: MapR user ticket not available! error = com.mapr.security.MutableInt@19cd7577
at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:205)
at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:110)
at com.dremio.dac.daemon.ZkServer$ZkEmbeddedServer.run(ZkServer.java:142)
at java.lang.Thread.run(Thread.java:748)

4

Make sure you you have specified MAPR_TICKET in dremio-env
https://docs.dremio.com/deployment/yarn-deployment.html?h=dremio-env

5

Thanks alot. Dremio is running now. But tried opening in 9047 port but Dremio UI is not loading. Can you please let me know where we can get the port number in Dremio

ps -ef | grep dremio
dremio 30821 1 6 15:07 ? 00:00:32 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el6_10.x86_64/jre/bin/java -Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler -Djava.library.path=/opt/dremio/lib -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/dremio/server.gc -Ddremio.log.path=/var/log/dremio -Xmx4096m -XX:MaxDirectMemorySize=8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/dremio -Dio.netty.maxDirectMemory=0 -DMAPR_IMPALA_RA_THROTTLE -DMAPR_MAX_RA_STREAMS=400 -cp /etc/dremio:/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/dremio/jars/:/opt/dremio/jars/ext/:/opt/dremio/jars/3rdparty/* com.dremio.dac.daemon.DremioDaemon dremio start

6

Could you look at the logs: /var/log/dremio/server.log and /var/log/dremio/server.out

7

PFB attached log files.

server.zip (8.9 KB)

8

Looks like process is stuck somewhere. Try to take jstack of the Dremio process may be?

9

Hi,
Is it possible to have multiple version of dremio to be executed at same time?
When i tried, i am getting error
Dremio is exiting. Failure while starting services.
java.lang.IllegalStateException: KVStore has an older version (3.3.2-201908142136370993-d60145d) than the server (4.0.0-201909121834570395-c7a5071), please run the upgrade tool first

10

@mmohan

The current version of Dremio software does not match with the KV store that contains our internal metadata. As the error suggests, it looks like the software is at a higher version so kindly run the “dremio-admin upgrade” command and then start Dremio

11

any resolution on this issue ? I have similar issue.

Dremio is exiting. Failure while starting services.
java.lang.IllegalStateException: KVStore has an older version (21.2.0-202205262146080444-038d6d1b) than the server (22.1.1-202208230402290397-a7010f28), please run the upgrade tool first
at com.google.common.base.Preconditions.checkState(Preconditions.java:821)
at com.dremio.dac.daemon.DremioDaemon$AutoUpgrade.ensureUpgradeSupported(DremioDaemon.java:67)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:228)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:186)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:99)

after dremio-admin upgrade , below error is showing.

Dremio is exiting. Failure while starting services.
com.dremio.services.credentials.SecretCredentialsException: Secret decryption encounters exception
at com.dremio.services.credentials.LocalSecretsStore.decrypt(LocalSecretsStore.java:128)
at com.dremio.services.credentials.LocalSecretsStore.doLookup(LocalSecretsStore.java:197)
at com.dremio.services.credentials.SecretCredentialsProvider.doLookup(SecretCredentialsProvider.java:57)
at com.dremio.services.credentials.AbstractSimpleCredentialsProvider.lookup(AbstractSimpleCredentialsProvider.java:43)
at com.dremio.services.credentials.SimpleCredentialsService.lookup(SimpleCredentialsService.java:106)
at com.dremio.extusr.ldap.LdapUserProvider.lookupSecretFromBindPassword(LdapUserProvider.java:257)
at com.dremio.extusr.ldap.LdapUserProvider.start(LdapUserProvider.java:239)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:88)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:196)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:202)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:104)
Suppressed: java.lang.NullPointerException
at com.dremio.service.orphanagecleaner.OrphanageCleanerService.close(OrphanageCleanerService.java:124)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.close(SingletonRegistry.java:130)
at com.dremio.common.AutoCloseables.close(AutoCloseables.java:139)
at com.dremio.service.ServiceRegistry.close(ServiceRegistry.java:96)
at com.dremio.service.SingletonRegistry.close(SingletonRegistry.java:90)
at com.dremio.common.AutoCloseables.close(AutoCloseables.java:139)
at com.dremio.common.AutoCloseables.close(AutoCloseables.java:76)
at com.dremio.dac.daemon.DACDaemon.close(DACDaemon.java:296)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:103)
Suppressed: java.lang.NullPointerException
at com.dremio.provision.service.ProvisioningServiceImpl.close(ProvisioningServiceImpl.java:512)
… 8 more
Caused by: javax.crypto.AEADBadTagException: Tag mismatch!
at com.sun.crypto.provider.GaloisCounterMode.decryptFinal(GaloisCounterMode.java:620)
at com.sun.crypto.provider.CipherCore.finalNoPadding(CipherCore.java:1116)
at com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1053)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:853)
at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
at javax.crypto.Cipher.doFinal(Cipher.java:2168)
at com.dremio.services.credentials.LocalSecretsStore.decrypt(LocalSecretsStore.java:125)
… 12 more

12

@saimanne Did you have a new version of Dremio installed?

Did you do dremio-admin upgrade then start Dremio?

https://docs.dremio.com/software/deployment/standalone/standalone-install-upgrade/