Urgent: Dremio will not start after upgrade to 3.2.0

summersmd · May 15, 2019, 2:51pm

even though I have used the dremio-admin upgrade command - which completed successfully, dremio will not start and I’m seeing the following error in the server.out file:

Dremio is exiting. Failure while starting services.
java.lang.IllegalStateException: KVStore has an older version (3.1.11-201904261857420193-c674472) than the server (3.2.0-201905102005330382-0598733), please run the upgrade tool first
at com.google.common.base.Preconditions.checkState(Preconditions.java:721)
at com.dremio.dac.daemon.DremioDaemon$AutoUpgrade.ensureUpgradeSupported(DremioDaemon.java:67)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:178)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:160)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:97)

and the following in server.log:

019-05-15 14:52:24,119 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
java.lang.IllegalStateException: KVStore has an older version (3.1.11-201904261857420193-c674472) than the server (3.2.0-201905102005330382-0598733), please run the upgrade tool first
at com.google.common.base.Preconditions.checkState(Preconditions.java:721) ~[guava-20.0.jar:na]
at com.dremio.dac.daemon.DremioDaemon$AutoUpgrade.ensureUpgradeSupported(DremioDaemon.java:67) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:178) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:160) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:97) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]

ben · May 15, 2019, 2:59pm

@summersmd, are you absolutely sure your are pointing Dremio to the correct db? What do you have for paths.db in dremio.conf

summersmd · May 15, 2019, 3:04pm

I’m sure I’m pointing to the correct db. It’s the same db we’ve been using and the configuration has not changed.

paths: {

the local path for dremio to store data.

local: “/app/dremio/data”

the distributed path Dremio data including job results, downloads, uploads, etc

dist: “pdfs://”${paths.local}"/pdfs"

location for catalog database (if master node)

db: ${paths.local}/db,

spilling: [${paths.local}/spill]

storage area for the accelerator cache.

accelerator: ${paths.dist}/accelerator

staging area for json and csv ui downloads

downloads: ${paths.dist}/downloads

stores uploaded data associated with user home directories

uploads: ${paths.dist}/uploads

stores data associated with the job results cache.

results: ${paths.dist}/results

shared scratch space for creation of tables.

scratch: ${paths.dist}/scratch
}

summersmd · May 15, 2019, 3:11pm

I reran ./dremio-admin upgrade as the dremio user account (instead of root) and now I’m getting a new error in server.out:

Dremio is exiting. Failure while starting services.
java.lang.NoSuchMethodError: org.eclipse.jetty.http.pathmap.PathMappings.put(Lorg/eclipse/jetty/http/pathmap/PathSpec;Ljava/lang/Object;)Z
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1421)
at org.eclipse.jetty.servlet.ServletHandler.doStart(ServletHandler.java:165)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:123)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:913)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:365)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.server.Server.start(Server.java:415)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:382)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at com.dremio.dac.server.WebServer.start(WebServer.java:257)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102)

summersmd · May 15, 2019, 3:11pm

and the following in server.log:

2019-05-15 15:11:17,755 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
java.lang.NoSuchMethodError: org.eclipse.jetty.http.pathmap.PathMappings.put(Lorg/eclipse/jetty/http/pathmap/PathSpec;Ljava/lang/Object;)Z
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1421) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.servlet.ServletHandler.doStart(ServletHandler.java:165) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:123) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:913) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:365) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.Server.start(Server.java:415) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.Server.doStart(Server.java:382) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at com.dremio.dac.server.WebServer.start(WebServer.java:257) ~[dremio-dac-backend-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137) ~[dremio-common-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74) ~[dremio-common-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33) ~[dremio-common-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183) ~[dremio-dac-backend-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189) ~[dremio-dac-backend-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]

balaji.ramaswamy · May 15, 2019, 5:55pm

@summersmd

What version of JDK are you using? Also who owns the Dremio files? Is it Dremio or root?

Thanks
@balaji.ramaswamy

summersmd · May 15, 2019, 7:04pm

@balaji.ramaswamy dremio owns the files. Using java 1.8.0_211.

balaji.ramaswamy · May 15, 2019, 7:18pm

Hi @summersmd

Is this a stand alone node? If not, did we bring down the executors and upgrade the software there too?

Thanks
@balaji.ramaswamy

summersmd · May 15, 2019, 7:22pm

This is a standalone node.

balaji.ramaswamy · May 15, 2019, 7:25pm

Ok to recap

JDK 1.8 running Dremio 3.1
Upgraded software to 3.2 also ran dremio-admin upgrade?
Now Dremio is not starting.

If this is the case kindly send us the server.log

Thanks
@balaji.ramaswamy

summersmd · May 15, 2019, 7:32pm

Confirmed:
JDK 1.8 running Dremio 3.1
Upgraded software to 3.2 also ran dremio-admin upgrade?
Now Dremio is not starting.

balaji.ramaswamy · May 15, 2019, 7:43pm

I see the below. Can you please do a df -h from the mount rocksDB is on?

2019-05-15 19:29:21,974 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
java.lang.ExceptionInInitializerError: null
at org.rocksdb.Options.(Options.java:25) ~[rocksdbjni-5.14.2.jar:na]
at com.dremio.datastore.ByteStoreManager.start(ByteStoreManager.java:201) ~[dremio-services-datastore-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.datastore.CoreStoreProviderImpl.start(CoreStoreProviderImpl.java:195) ~[dremio-services-datastore-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.datastore.LocalKVStoreProvider.start(LocalKVStoreProvider.java:157) ~[dremio-services-datastore-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:158) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:97) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
Caused by: java.lang.RuntimeException: Unable to load the RocksDB shared libraryjava.io.IOException: No space left on device
at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:67) ~[rocksdbjni-5.14.2.jar:na]
at org.rocksdb.RocksDB.(RocksDB.java:35) ~[rocksdbjni-5.14.2.jar:na]
… 6 common frames omitted

summersmd · May 15, 2019, 7:49pm

How is the location of the rocksdb configured? I see our /tmp directory is 100% full and it looks like the rocksdb is saved there, but I’m not sure why it would be saved there. It looks like I need to change the location for rocksdb to something other than /tmp.

balaji.ramaswamy · May 15, 2019, 8:16pm

Here is the documentation

https://docs.dremio.com/advanced-administration/dremio-conf.html

paths: {

  # the local path for dremio to store data.
  local: ${DREMIO_HOME}"/data"

summersmd · May 15, 2019, 8:28pm

In our dremio.conf file we have:

paths: {
local: “/app/dremio/data”

which is the path we’ve used for all of our previous dremio installs and we’ve never had this issue before.

for some reason, /tmp is being used.

balaji.ramaswamy · May 15, 2019, 9:04pm

Can you please send df -h /tmp and ls -ltrh /tmp outputs?

summersmd · May 16, 2019, 2:45pm

I think I have fixed the issue with /tmp running out of space, however now I’m now getting a new error message. I’m growing concerned because I have a proof of concept demo on Monday. The new error message I’m seeing is:

Dremio is exiting. Failure while starting services.
java.lang.NoSuchMethodError: org.eclipse.jetty.http.pathmap.PathMappings.put(Lorg/eclipse/jetty/http/pathmap/PathSpec;Ljava/lang/Object;)Z
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1421)
at org.eclipse.jetty.servlet.ServletHandler.doStart(ServletHandler.java:165)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:123)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:913)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:365)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.server.Server.start(Server.java:415)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:382)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at com.dremio.dac.server.WebServer.start(WebServer.java:257)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102)

summersmd · May 16, 2019, 3:00pm

I stand corrected, the longer I leave dremio running, the more files it writes to /tmp.

df -h /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-tmp 2.0G 2.0G 0 100% /tmp

ls -lrth /tmp | grep dremio
drwxr-xr-x 2 dremio dremio 6 May 7 01:38 dremio-support
-rw-r–r-- 1 dremio dremio 146K May 15 16:38 irhelpers.bc30552ffd-1b09-4457-89fe-e98ae0fb3abb
-rw-r–r-- 1 dremio dremio 7.9M May 15 16:42 librocksdbjni2474945149489956128.so
-rw-r–r-- 1 dremio dremio 36M May 15 16:43 libgandiva_jni.so1819c659-5827-447e-b304-42ce168ef62c
-rw-r–r-- 1 dremio dremio 146K May 15 16:43 irhelpers.bc88666329-edd4-47a9-9f5d-ac61bdb14e61
-rw-r–r-- 1 dremio dremio 7.9M May 15 16:43 librocksdbjni4768622427663942288.so
-rw-r–r-- 1 dremio dremio 36M May 15 16:44 libgandiva_jni.so3fc95d50-fdd4-4a62-8390-251b4a007531
-rw-r–r-- 1 dremio dremio 146K May 15 16:44 irhelpers.bcceb9a66f-f7c9-4c95-80af-a17ae8566153
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:43 librocksdbjni5417623524275247120.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:45 libgandiva_jni.soc7b195ec-4a18-4c30-ae60-eab0617b7897
-rw-r–r-- 1 dremio dremio 146K May 16 10:45 irhelpers.bc2bab94f2-1ac8-45e7-866d-151bdf6373c9
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:46 librocksdbjni8384188306582098660.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:46 libgandiva_jni.so5a81702f-800c-456c-b0e6-da0ca2cee2d8
-rw-r–r-- 1 dremio dremio 146K May 16 10:46 irhelpers.bc7f23d312-91b5-4a61-92ed-134c33faa156
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:47 librocksdbjni4062246913097261205.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:48 libgandiva_jni.so8293c76e-f36f-4c49-98d9-6b1d53cd90fb
-rw-r–r-- 1 dremio dremio 146K May 16 10:48 irhelpers.bcec4f6f66-f9d5-40dc-8887-7738a30dc5fa
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:48 librocksdbjni6280709598633111714.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:49 libgandiva_jni.so18b92d7c-e1fa-4661-b606-459e50e63940
-rw-r–r-- 1 dremio dremio 146K May 16 10:49 irhelpers.bc5de7f42d-0d92-4c7d-9be0-86f3c079d4ac
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:49 librocksdbjni3667222257910104154.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:50 libgandiva_jni.so03f8c0aa-a4ff-4b0a-98d0-42afffd07c4c
-rw-r–r-- 1 dremio dremio 146K May 16 10:50 irhelpers.bcfaf206b8-2225-4c17-a530-90f70e2824d9
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:50 librocksdbjni228048964999189158.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:51 libgandiva_jni.so0297a3ef-5508-4bf4-ad7b-a18598cf8102
-rw-r–r-- 1 dremio dremio 146K May 16 10:51 irhelpers.bcde8ef139-06cf-4e7e-8d80-28dbbf6bd470
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:52 librocksdbjni7997952340478346314.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:52 libgandiva_jni.so27e55edb-498e-4617-a105-da7e38b3cfe3
-rw-r–r-- 1 dremio dremio 146K May 16 10:52 irhelpers.bc063085a0-a73b-4fa0-a4b1-0399526211e5
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:53 librocksdbjni3154744454342548913.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:53 libgandiva_jni.so9f84671d-1c87-4f40-bc5e-94187f779041
-rw-r–r-- 1 dremio dremio 146K May 16 10:53 irhelpers.bcb4545f10-ccc8-409d-9503-e63eae7f51ec
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:54 librocksdbjni3331418003058889195.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:54 libgandiva_jni.so5a35ccd7-9b1d-49f1-a54d-70f8efe2d87a
-rw-r–r-- 1 dremio dremio 146K May 16 10:54 irhelpers.bc5ed0ca94-e8c0-4758-99a7-5e59348b6742
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:55 librocksdbjni3780990704380170840.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:55 libgandiva_jni.so020eee57-a951-4cb9-9967-55cb46a34c03
-rw-r–r-- 1 dremio dremio 146K May 16 10:55 irhelpers.bcee3b5181-a029-4833-9a08-a49685845815
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:56 librocksdbjni8770384686193740154.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:56 libgandiva_jni.sof8ae024c-a582-4ce6-aa01-93046bc3dbc4
-rw-r–r-- 1 dremio dremio 146K May 16 10:56 irhelpers.bc12f2b8a7-10f2-4a6e-bc6d-11f00c4bb569
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:57 librocksdbjni2868343319686299212.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:58 libgandiva_jni.soe82390f7-0e26-4ad2-ad30-c08fd7f84baa
-rw-r–r-- 1 dremio dremio 146K May 16 10:58 irhelpers.bceee56ef3-e867-4b9d-bc3e-bf17ee51d4e0
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:58 librocksdbjni5430611665186786377.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:59 libgandiva_jni.so194879e8-eb3f-484a-9392-a3abcbe3f94c
-rw-r–r-- 1 dremio dremio 146K May 16 10:59 irhelpers.bc22c67afc-ff55-4f6d-a484-2156e8bcff84
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:59 librocksdbjni16987508368763531.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:00 libgandiva_jni.so8fd88c80-8948-48b1-9925-59c3ba97f432
-rw-r–r-- 1 dremio dremio 146K May 16 11:00 irhelpers.bcbb147a92-6d85-4682-a1c6-3ffea6ac3e6c
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:00 librocksdbjni8495180989958949497.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:01 libgandiva_jni.sofc09a72c-7052-4848-967c-ecdb7e597b07
-rw-r–r-- 1 dremio dremio 146K May 16 11:01 irhelpers.bca33b70c7-c4b8-4233-b131-4af095f30d5d
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:01 librocksdbjni2111038461817832352.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:02 libgandiva_jni.so05a21594-a1fb-419b-bfc7-4e7d455c91c9
-rw-r–r-- 1 dremio dremio 146K May 16 11:02 irhelpers.bc6354810a-1de8-480f-86ca-524ee93594f4
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:02 librocksdbjni4794141259125900654.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:03 libgandiva_jni.so3910938e-ba3c-4269-9ec5-46564568e1d5
-rw-r–r-- 1 dremio dremio 146K May 16 11:03 irhelpers.bce860641b-a6ad-49e3-940c-949a63c3b577
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:04 librocksdbjni6008862212927600634.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:04 libgandiva_jni.so1e37c37d-51d8-4357-9d7d-c78ca2a61936
-rw-r–r-- 1 dremio dremio 146K May 16 11:04 irhelpers.bc9e806f79-54f6-4bce-ae3f-f6dc830eacfc
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:05 librocksdbjni5438135808316540718.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:05 libgandiva_jni.so92b62f55-0580-4831-8e39-397a40e08848
-rw-r–r-- 1 dremio dremio 146K May 16 11:05 irhelpers.bcb3ba39e6-21e1-46b4-84a5-3d1a00f7e00d
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:06 librocksdbjni7790380156454323906.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:07 libgandiva_jni.sof664fc57-4269-4510-869a-376bd7baa5cd
-rw-r–r-- 1 dremio dremio 146K May 16 11:07 irhelpers.bcdc725a56-dc67-4529-af4e-fc76a1e76b87
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:07 librocksdbjni7635437086162451889.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:08 libgandiva_jni.so6c7cb961-608a-46d8-83d9-0e94c8c5b433
-rw-r–r-- 1 dremio dremio 146K May 16 11:08 irhelpers.bcf11d9569-352b-4d1d-a719-5b236c1bfb99
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:08 librocksdbjni6518991588091194938.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:09 libgandiva_jni.sofb81071d-4920-4a62-8b00-5507ed7db359
-rw-r–r-- 1 dremio dremio 146K May 16 11:09 irhelpers.bc9c3500e2-a299-448d-b000-9c1a53f0b4ee
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:09 librocksdbjni5389221458865006443.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:10 libgandiva_jni.so4599fbac-32a7-4eff-b833-1c4e59bdc3a3
-rw-r–r-- 1 dremio dremio 146K May 16 11:10 irhelpers.bcec2dea2b-e165-4bd0-a8d9-163ce71736e6
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:11 librocksdbjni97702939113759583.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:11 libgandiva_jni.soe593f0cb-b1d9-40f3-b6ab-65265d936eca
-rw-r–r-- 1 dremio dremio 146K May 16 11:11 irhelpers.bcb0f3fa6e-a6bd-4ca7-be2d-b5f489586869
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:12 librocksdbjni1096290942753668021.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:12 libgandiva_jni.soded70803-a06b-4e78-bd22-3f41a84dc88c
-rw-r–r-- 1 dremio dremio 146K May 16 11:12 irhelpers.bc207726a7-b32b-49a2-8ecd-d40231c87b46
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:13 librocksdbjni5369465178013541165.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:13 libgandiva_jni.so4b92956d-e86e-4bbd-8105-395a0876fb85
-rw-r–r-- 1 dremio dremio 146K May 16 11:13 irhelpers.bc76125676-b8f8-489a-b119-66ef767ba1a4
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:14 librocksdbjni915796185479156281.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:14 libgandiva_jni.soc1ae666a-69df-4056-967f-d260e6b34a12
-rw-r–r-- 1 dremio dremio 146K May 16 11:14 irhelpers.bc8cf0d620-324d-451d-afbc-f483e2157386
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:15 librocksdbjni100666521919505226.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:15 libgandiva_jni.so173963f2-7cac-4213-b10b-3b46687a4b58
-rw-r–r-- 1 dremio dremio 146K May 16 11:15 irhelpers.bc736c3d0f-c392-4d43-90ce-259b91662b2a
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:16 librocksdbjni6884246809784225427.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:16 libgandiva_jni.so92af4ba8-56b5-419f-8432-8bf32d025149
-rw-r–r-- 1 dremio dremio 146K May 16 11:16 irhelpers.bc6eb51d99-522b-4cbc-bd35-e08404c6861c
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:17 librocksdbjni4443737537568495709.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:17 libgandiva_jni.so5405510e-c62f-4448-8e47-11c3ec97b35d
-rw-r–r-- 1 dremio dremio 146K May 16 11:17 irhelpers.bc9d9cd87a-5ed8-4828-9b79-cc625485e6ba
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:18 librocksdbjni531952699970304132.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:18 libgandiva_jni.so9741607f-817b-4672-862b-afb9b6e9d188
-rw-r–r-- 1 dremio dremio 146K May 16 11:18 irhelpers.bcb9dedf33-f116-42fe-ae64-2c63d1216d33
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:19 librocksdbjni1208636884817099782.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:20 libgandiva_jni.so8b9af1e1-36e8-4b73-a8a1-3baf0329fe09
-rw-r–r-- 1 dremio dremio 146K May 16 11:20 irhelpers.bca7bfeb1c-e3a1-484a-9540-ca92a93de909
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:20 librocksdbjni2430962552333391106.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:21 libgandiva_jni.sob4f42290-dc46-4a86-b7b5-424ebe122a24
-rw-r–r-- 1 dremio dremio 146K May 16 11:21 irhelpers.bcd49a9f78-9bd1-4739-96a3-ab7df22ce150
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:21 librocksdbjni1362561622445855994.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:22 libgandiva_jni.so532a9ada-713a-4e5f-a446-8cef95a02e84
-rw-r–r-- 1 dremio dremio 146K May 16 11:22 irhelpers.bc98952b35-dee6-438f-88ad-416d0f40b1de
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:22 librocksdbjni4562727618918815357.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:23 libgandiva_jni.so6b67f25d-8d5c-4301-bd46-10079890279b
-rw-r–r-- 1 dremio dremio 146K May 16 11:23 irhelpers.bc5a877d8d-1029-412b-ad77-7506b1c7bf33
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:23 librocksdbjni6179365421258594994.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:24 libgandiva_jni.so38d3c3f6-4262-48a4-a15c-5c2f185e5385
-rw-r–r-- 1 dremio dremio 146K May 16 11:24 irhelpers.bccaa8fda4-ee1d-42c9-b62a-d6c1266d9964
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:24 librocksdbjni1775497409614260618.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:25 libgandiva_jni.so20ddb292-20a8-4d88-a3ea-45cf2624948c
-rw-r–r-- 1 dremio dremio 146K May 16 11:25 irhelpers.bc8735934d-5919-4fa7-bdb1-8befa94690c2
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:25 librocksdbjni8850648580910683768.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:26 libgandiva_jni.soef27dee0-2b40-4055-a412-371eeee7ef14
-rw-r–r-- 1 dremio dremio 146K May 16 11:26 irhelpers.bcc513b672-3338-4efa-8b4b-8ad62b4ec8f3
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:26 librocksdbjni1128199500014421850.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:27 libgandiva_jni.so072f09b5-134a-4a8a-807d-d55d0ac3c9b2
-rw-r–r-- 1 dremio dremio 146K May 16 11:27 irhelpers.bc22883e51-f659-4d97-951e-bdce94e5b5b8
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:27 librocksdbjni4660007239191371976.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:28 libgandiva_jni.so51279231-e264-4ccd-8d08-1735104cd143
-rw-r–r-- 1 dremio dremio 146K May 16 11:28 irhelpers.bc17f48029-1feb-46f7-ac8a-0763a9ca0e5f
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:29 librocksdbjni3808416489901804549.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:29 libgandiva_jni.so0fc47bab-8ebc-4a7c-9fe9-9e6bb0110589
-rw-r–r-- 1 dremio dremio 146K May 16 11:29 irhelpers.bc76ccd155-39d1-4133-aeed-8de6d2c051ff
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:29 librocksdbjni3002667637748772840.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:30 libgandiva_jni.so64f06d6e-57a8-4a99-be35-927b6384a7ef
-rw-r–r-- 1 dremio dremio 146K May 16 11:30 irhelpers.bcf84b7ec3-a172-4331-abba-6867bbfa4235
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:31 librocksdbjni8036948367534165272.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:31 libgandiva_jni.so08062516-274b-47e8-ac24-58af5a83c354
-rw-r–r-- 1 dremio dremio 146K May 16 11:31 irhelpers.bc75bd17d3-85a1-4a15-8cb0-55c5c29d12a2
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:32 librocksdbjni3320584694528516618.so
-rw-r–r-- 1 dremio dremio 27M May 16 11:32 libgandiva_jni.so9faadb7b-896b-48fc-9076-fa0d73d24280
-rw-r–r-- 1 dremio dremio 0 May 16 11:32 libgandiva_jni.so96d2f14f-21a5-471d-8a6f-fbdcef55272a
drwxr-xr-x 2 dremio dremio 108 May 16 11:33 jna–1323653796
-rw-r–r-- 1 dremio dremio 0 May 16 11:33 librocksdbjni7547400547210792976.so
-rw-r–r-- 1 dremio dremio 0 May 16 11:33 librocksdbjni2354933948589969896.so
-rw-r–r-- 1 dremio dremio 0 May 16 11:34 librocksdbjni2888830322657183489.so
-rw-r–r-- 1 dremio dremio 0 May 16 11:34 librocksdbjni5673017548043232479.so
drwxr-xr-x 2 dremio dremio 20 May 16 11:34 hsperfdata_dremio

hakim · May 16, 2019, 5:04pm

This suggests Dremio is trying to use the wrong jars. Can you do a recursive ls of your Dremio’s jars folder and post it here ? Thanks!
Also how did you install Dremio, rpm, tar ?

summersmd · May 16, 2019, 5:06pm

I installed via RPM.

Topic		Replies	Views
Upgrading Dremio to 3.1 Error	2	1756	February 5, 2019
Unable to start Dremio in UNIX - KVStore has an older version	11	3352	February 9, 2023
Upgrade dremio (KVStore) running on docker compose	6	26	October 19, 2024
Which files to copy from our old install?	4	1305	June 3, 2019
Dremio won't start after 3.2 upgrade: KVStore has an older version	7	1856	May 23, 2019

Urgent: Dremio will not start after upgrade to 3.2.0

the local path for dremio to store data.

the distributed path Dremio data including job results, downloads, uploads, etc

location for catalog database (if master node)

storage area for the accelerator cache.

staging area for json and csv ui downloads

stores uploaded data associated with user home directories

stores data associated with the job results cache.

shared scratch space for creation of tables.

Related topics