Urgent: Dremio will not start after upgrade to 3.2.0

even though I have used the dremio-admin upgrade command - which completed successfully, dremio will not start and I’m seeing the following error in the server.out file:

Dremio is exiting. Failure while starting services.
java.lang.IllegalStateException: KVStore has an older version (3.1.11-201904261857420193-c674472) than the server (3.2.0-201905102005330382-0598733), please run the upgrade tool first
at com.google.common.base.Preconditions.checkState(Preconditions.java:721)
at com.dremio.dac.daemon.DremioDaemon$AutoUpgrade.ensureUpgradeSupported(DremioDaemon.java:67)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:178)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:160)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:97)

and the following in server.log:

019-05-15 14:52:24,119 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
java.lang.IllegalStateException: KVStore has an older version (3.1.11-201904261857420193-c674472) than the server (3.2.0-201905102005330382-0598733), please run the upgrade tool first
at com.google.common.base.Preconditions.checkState(Preconditions.java:721) ~[guava-20.0.jar:na]
at com.dremio.dac.daemon.DremioDaemon$AutoUpgrade.ensureUpgradeSupported(DremioDaemon.java:67) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:178) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:160) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:97) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]

@summersmd, are you absolutely sure your are pointing Dremio to the correct db? What do you have for paths.db in dremio.conf

I’m sure I’m pointing to the correct db. It’s the same db we’ve been using and the configuration has not changed.

paths: {

the local path for dremio to store data.

local: “/app/dremio/data”

the distributed path Dremio data including job results, downloads, uploads, etc

dist: “pdfs://”${paths.local}"/pdfs"

location for catalog database (if master node)

db: ${paths.local}/db,

spilling: [${paths.local}/spill]

storage area for the accelerator cache.

accelerator: ${paths.dist}/accelerator

staging area for json and csv ui downloads

downloads: ${paths.dist}/downloads

stores uploaded data associated with user home directories

uploads: ${paths.dist}/uploads

stores data associated with the job results cache.

results: ${paths.dist}/results

shared scratch space for creation of tables.

scratch: ${paths.dist}/scratch
}

I reran ./dremio-admin upgrade as the dremio user account (instead of root) and now I’m getting a new error in server.out:

Dremio is exiting. Failure while starting services.
java.lang.NoSuchMethodError: org.eclipse.jetty.http.pathmap.PathMappings.put(Lorg/eclipse/jetty/http/pathmap/PathSpec;Ljava/lang/Object;)Z
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1421)
at org.eclipse.jetty.servlet.ServletHandler.doStart(ServletHandler.java:165)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:123)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:913)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:365)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.server.Server.start(Server.java:415)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:382)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at com.dremio.dac.server.WebServer.start(WebServer.java:257)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102)

and the following in server.log:

2019-05-15 15:11:17,755 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
java.lang.NoSuchMethodError: org.eclipse.jetty.http.pathmap.PathMappings.put(Lorg/eclipse/jetty/http/pathmap/PathSpec;Ljava/lang/Object;)Z
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1421) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.servlet.ServletHandler.doStart(ServletHandler.java:165) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:123) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:913) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:365) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.Server.start(Server.java:415) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.Server.doStart(Server.java:382) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at com.dremio.dac.server.WebServer.start(WebServer.java:257) ~[dremio-dac-backend-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137) ~[dremio-common-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74) ~[dremio-common-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33) ~[dremio-common-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183) ~[dremio-dac-backend-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189) ~[dremio-dac-backend-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]

@summersmd

What version of JDK are you using? Also who owns the Dremio files? Is it Dremio or root?

Thanks
@balaji.ramaswamy

@balaji.ramaswamy dremio owns the files. Using java 1.8.0_211.

Hi @summersmd

Is this a stand alone node? If not, did we bring down the executors and upgrade the software there too?

Thanks
@balaji.ramaswamy

This is a standalone node.

Ok to recap

JDK 1.8 running Dremio 3.1
Upgraded software to 3.2 also ran dremio-admin upgrade?
Now Dremio is not starting.

If this is the case kindly send us the server.log

Thanks
@balaji.ramaswamy

Confirmed:
JDK 1.8 running Dremio 3.1
Upgraded software to 3.2 also ran dremio-admin upgrade?
Now Dremio is not starting.

I see the below. Can you please do a df -h from the mount rocksDB is on?

2019-05-15 19:29:21,974 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
java.lang.ExceptionInInitializerError: null
at org.rocksdb.Options.(Options.java:25) ~[rocksdbjni-5.14.2.jar:na]
at com.dremio.datastore.ByteStoreManager.start(ByteStoreManager.java:201) ~[dremio-services-datastore-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.datastore.CoreStoreProviderImpl.start(CoreStoreProviderImpl.java:195) ~[dremio-services-datastore-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.datastore.LocalKVStoreProvider.start(LocalKVStoreProvider.java:157) ~[dremio-services-datastore-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:158) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:97) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
Caused by: java.lang.RuntimeException: Unable to load the RocksDB shared libraryjava.io.IOException: No space left on device
at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:67) ~[rocksdbjni-5.14.2.jar:na]
at org.rocksdb.RocksDB.(RocksDB.java:35) ~[rocksdbjni-5.14.2.jar:na]
… 6 common frames omitted

How is the location of the rocksdb configured? I see our /tmp directory is 100% full and it looks like the rocksdb is saved there, but I’m not sure why it would be saved there. It looks like I need to change the location for rocksdb to something other than /tmp.

Here is the documentation

https://docs.dremio.com/advanced-administration/dremio-conf.html

paths: {

  # the local path for dremio to store data.
  local: ${DREMIO_HOME}"/data"

In our dremio.conf file we have:

paths: {
local: “/app/dremio/data”

which is the path we’ve used for all of our previous dremio installs and we’ve never had this issue before.

for some reason, /tmp is being used.

Can you please send df -h /tmp and ls -ltrh /tmp outputs?

I think I have fixed the issue with /tmp running out of space, however now I’m now getting a new error message. I’m growing concerned because I have a proof of concept demo on Monday. The new error message I’m seeing is:

Dremio is exiting. Failure while starting services.
java.lang.NoSuchMethodError: org.eclipse.jetty.http.pathmap.PathMappings.put(Lorg/eclipse/jetty/http/pathmap/PathSpec;Ljava/lang/Object;)Z
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1421)
at org.eclipse.jetty.servlet.ServletHandler.doStart(ServletHandler.java:165)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:123)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:913)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:365)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.server.Server.start(Server.java:415)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:382)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at com.dremio.dac.server.WebServer.start(WebServer.java:257)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102)

I stand corrected, the longer I leave dremio running, the more files it writes to /tmp.

df -h /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-tmp 2.0G 2.0G 0 100% /tmp


ls -lrth /tmp | grep dremio
drwxr-xr-x 2 dremio dremio 6 May 7 01:38 dremio-support
-rw-r–r-- 1 dremio dremio 146K May 15 16:38 irhelpers.bc30552ffd-1b09-4457-89fe-e98ae0fb3abb
-rw-r–r-- 1 dremio dremio 7.9M May 15 16:42 librocksdbjni2474945149489956128.so
-rw-r–r-- 1 dremio dremio 36M May 15 16:43 libgandiva_jni.so1819c659-5827-447e-b304-42ce168ef62c
-rw-r–r-- 1 dremio dremio 146K May 15 16:43 irhelpers.bc88666329-edd4-47a9-9f5d-ac61bdb14e61
-rw-r–r-- 1 dremio dremio 7.9M May 15 16:43 librocksdbjni4768622427663942288.so
-rw-r–r-- 1 dremio dremio 36M May 15 16:44 libgandiva_jni.so3fc95d50-fdd4-4a62-8390-251b4a007531
-rw-r–r-- 1 dremio dremio 146K May 15 16:44 irhelpers.bcceb9a66f-f7c9-4c95-80af-a17ae8566153
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:43 librocksdbjni5417623524275247120.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:45 libgandiva_jni.soc7b195ec-4a18-4c30-ae60-eab0617b7897
-rw-r–r-- 1 dremio dremio 146K May 16 10:45 irhelpers.bc2bab94f2-1ac8-45e7-866d-151bdf6373c9
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:46 librocksdbjni8384188306582098660.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:46 libgandiva_jni.so5a81702f-800c-456c-b0e6-da0ca2cee2d8
-rw-r–r-- 1 dremio dremio 146K May 16 10:46 irhelpers.bc7f23d312-91b5-4a61-92ed-134c33faa156
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:47 librocksdbjni4062246913097261205.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:48 libgandiva_jni.so8293c76e-f36f-4c49-98d9-6b1d53cd90fb
-rw-r–r-- 1 dremio dremio 146K May 16 10:48 irhelpers.bcec4f6f66-f9d5-40dc-8887-7738a30dc5fa
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:48 librocksdbjni6280709598633111714.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:49 libgandiva_jni.so18b92d7c-e1fa-4661-b606-459e50e63940
-rw-r–r-- 1 dremio dremio 146K May 16 10:49 irhelpers.bc5de7f42d-0d92-4c7d-9be0-86f3c079d4ac
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:49 librocksdbjni3667222257910104154.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:50 libgandiva_jni.so03f8c0aa-a4ff-4b0a-98d0-42afffd07c4c
-rw-r–r-- 1 dremio dremio 146K May 16 10:50 irhelpers.bcfaf206b8-2225-4c17-a530-90f70e2824d9
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:50 librocksdbjni228048964999189158.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:51 libgandiva_jni.so0297a3ef-5508-4bf4-ad7b-a18598cf8102
-rw-r–r-- 1 dremio dremio 146K May 16 10:51 irhelpers.bcde8ef139-06cf-4e7e-8d80-28dbbf6bd470
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:52 librocksdbjni7997952340478346314.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:52 libgandiva_jni.so27e55edb-498e-4617-a105-da7e38b3cfe3
-rw-r–r-- 1 dremio dremio 146K May 16 10:52 irhelpers.bc063085a0-a73b-4fa0-a4b1-0399526211e5
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:53 librocksdbjni3154744454342548913.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:53 libgandiva_jni.so9f84671d-1c87-4f40-bc5e-94187f779041
-rw-r–r-- 1 dremio dremio 146K May 16 10:53 irhelpers.bcb4545f10-ccc8-409d-9503-e63eae7f51ec
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:54 librocksdbjni3331418003058889195.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:54 libgandiva_jni.so5a35ccd7-9b1d-49f1-a54d-70f8efe2d87a
-rw-r–r-- 1 dremio dremio 146K May 16 10:54 irhelpers.bc5ed0ca94-e8c0-4758-99a7-5e59348b6742
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:55 librocksdbjni3780990704380170840.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:55 libgandiva_jni.so020eee57-a951-4cb9-9967-55cb46a34c03
-rw-r–r-- 1 dremio dremio 146K May 16 10:55 irhelpers.bcee3b5181-a029-4833-9a08-a49685845815
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:56 librocksdbjni8770384686193740154.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:56 libgandiva_jni.sof8ae024c-a582-4ce6-aa01-93046bc3dbc4
-rw-r–r-- 1 dremio dremio 146K May 16 10:56 irhelpers.bc12f2b8a7-10f2-4a6e-bc6d-11f00c4bb569
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:57 librocksdbjni2868343319686299212.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:58 libgandiva_jni.soe82390f7-0e26-4ad2-ad30-c08fd7f84baa
-rw-r–r-- 1 dremio dremio 146K May 16 10:58 irhelpers.bceee56ef3-e867-4b9d-bc3e-bf17ee51d4e0
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:58 librocksdbjni5430611665186786377.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:59 libgandiva_jni.so194879e8-eb3f-484a-9392-a3abcbe3f94c
-rw-r–r-- 1 dremio dremio 146K May 16 10:59 irhelpers.bc22c67afc-ff55-4f6d-a484-2156e8bcff84
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:59 librocksdbjni16987508368763531.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:00 libgandiva_jni.so8fd88c80-8948-48b1-9925-59c3ba97f432
-rw-r–r-- 1 dremio dremio 146K May 16 11:00 irhelpers.bcbb147a92-6d85-4682-a1c6-3ffea6ac3e6c
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:00 librocksdbjni8495180989958949497.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:01 libgandiva_jni.sofc09a72c-7052-4848-967c-ecdb7e597b07
-rw-r–r-- 1 dremio dremio 146K May 16 11:01 irhelpers.bca33b70c7-c4b8-4233-b131-4af095f30d5d
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:01 librocksdbjni2111038461817832352.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:02 libgandiva_jni.so05a21594-a1fb-419b-bfc7-4e7d455c91c9
-rw-r–r-- 1 dremio dremio 146K May 16 11:02 irhelpers.bc6354810a-1de8-480f-86ca-524ee93594f4
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:02 librocksdbjni4794141259125900654.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:03 libgandiva_jni.so3910938e-ba3c-4269-9ec5-46564568e1d5
-rw-r–r-- 1 dremio dremio 146K May 16 11:03 irhelpers.bce860641b-a6ad-49e3-940c-949a63c3b577
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:04 librocksdbjni6008862212927600634.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:04 libgandiva_jni.so1e37c37d-51d8-4357-9d7d-c78ca2a61936
-rw-r–r-- 1 dremio dremio 146K May 16 11:04 irhelpers.bc9e806f79-54f6-4bce-ae3f-f6dc830eacfc
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:05 librocksdbjni5438135808316540718.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:05 libgandiva_jni.so92b62f55-0580-4831-8e39-397a40e08848
-rw-r–r-- 1 dremio dremio 146K May 16 11:05 irhelpers.bcb3ba39e6-21e1-46b4-84a5-3d1a00f7e00d
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:06 librocksdbjni7790380156454323906.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:07 libgandiva_jni.sof664fc57-4269-4510-869a-376bd7baa5cd
-rw-r–r-- 1 dremio dremio 146K May 16 11:07 irhelpers.bcdc725a56-dc67-4529-af4e-fc76a1e76b87
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:07 librocksdbjni7635437086162451889.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:08 libgandiva_jni.so6c7cb961-608a-46d8-83d9-0e94c8c5b433
-rw-r–r-- 1 dremio dremio 146K May 16 11:08 irhelpers.bcf11d9569-352b-4d1d-a719-5b236c1bfb99
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:08 librocksdbjni6518991588091194938.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:09 libgandiva_jni.sofb81071d-4920-4a62-8b00-5507ed7db359
-rw-r–r-- 1 dremio dremio 146K May 16 11:09 irhelpers.bc9c3500e2-a299-448d-b000-9c1a53f0b4ee
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:09 librocksdbjni5389221458865006443.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:10 libgandiva_jni.so4599fbac-32a7-4eff-b833-1c4e59bdc3a3
-rw-r–r-- 1 dremio dremio 146K May 16 11:10 irhelpers.bcec2dea2b-e165-4bd0-a8d9-163ce71736e6
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:11 librocksdbjni97702939113759583.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:11 libgandiva_jni.soe593f0cb-b1d9-40f3-b6ab-65265d936eca
-rw-r–r-- 1 dremio dremio 146K May 16 11:11 irhelpers.bcb0f3fa6e-a6bd-4ca7-be2d-b5f489586869
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:12 librocksdbjni1096290942753668021.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:12 libgandiva_jni.soded70803-a06b-4e78-bd22-3f41a84dc88c
-rw-r–r-- 1 dremio dremio 146K May 16 11:12 irhelpers.bc207726a7-b32b-49a2-8ecd-d40231c87b46
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:13 librocksdbjni5369465178013541165.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:13 libgandiva_jni.so4b92956d-e86e-4bbd-8105-395a0876fb85
-rw-r–r-- 1 dremio dremio 146K May 16 11:13 irhelpers.bc76125676-b8f8-489a-b119-66ef767ba1a4
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:14 librocksdbjni915796185479156281.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:14 libgandiva_jni.soc1ae666a-69df-4056-967f-d260e6b34a12
-rw-r–r-- 1 dremio dremio 146K May 16 11:14 irhelpers.bc8cf0d620-324d-451d-afbc-f483e2157386
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:15 librocksdbjni100666521919505226.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:15 libgandiva_jni.so173963f2-7cac-4213-b10b-3b46687a4b58
-rw-r–r-- 1 dremio dremio 146K May 16 11:15 irhelpers.bc736c3d0f-c392-4d43-90ce-259b91662b2a
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:16 librocksdbjni6884246809784225427.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:16 libgandiva_jni.so92af4ba8-56b5-419f-8432-8bf32d025149
-rw-r–r-- 1 dremio dremio 146K May 16 11:16 irhelpers.bc6eb51d99-522b-4cbc-bd35-e08404c6861c
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:17 librocksdbjni4443737537568495709.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:17 libgandiva_jni.so5405510e-c62f-4448-8e47-11c3ec97b35d
-rw-r–r-- 1 dremio dremio 146K May 16 11:17 irhelpers.bc9d9cd87a-5ed8-4828-9b79-cc625485e6ba
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:18 librocksdbjni531952699970304132.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:18 libgandiva_jni.so9741607f-817b-4672-862b-afb9b6e9d188
-rw-r–r-- 1 dremio dremio 146K May 16 11:18 irhelpers.bcb9dedf33-f116-42fe-ae64-2c63d1216d33
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:19 librocksdbjni1208636884817099782.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:20 libgandiva_jni.so8b9af1e1-36e8-4b73-a8a1-3baf0329fe09
-rw-r–r-- 1 dremio dremio 146K May 16 11:20 irhelpers.bca7bfeb1c-e3a1-484a-9540-ca92a93de909
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:20 librocksdbjni2430962552333391106.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:21 libgandiva_jni.sob4f42290-dc46-4a86-b7b5-424ebe122a24
-rw-r–r-- 1 dremio dremio 146K May 16 11:21 irhelpers.bcd49a9f78-9bd1-4739-96a3-ab7df22ce150
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:21 librocksdbjni1362561622445855994.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:22 libgandiva_jni.so532a9ada-713a-4e5f-a446-8cef95a02e84
-rw-r–r-- 1 dremio dremio 146K May 16 11:22 irhelpers.bc98952b35-dee6-438f-88ad-416d0f40b1de
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:22 librocksdbjni4562727618918815357.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:23 libgandiva_jni.so6b67f25d-8d5c-4301-bd46-10079890279b
-rw-r–r-- 1 dremio dremio 146K May 16 11:23 irhelpers.bc5a877d8d-1029-412b-ad77-7506b1c7bf33
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:23 librocksdbjni6179365421258594994.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:24 libgandiva_jni.so38d3c3f6-4262-48a4-a15c-5c2f185e5385
-rw-r–r-- 1 dremio dremio 146K May 16 11:24 irhelpers.bccaa8fda4-ee1d-42c9-b62a-d6c1266d9964
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:24 librocksdbjni1775497409614260618.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:25 libgandiva_jni.so20ddb292-20a8-4d88-a3ea-45cf2624948c
-rw-r–r-- 1 dremio dremio 146K May 16 11:25 irhelpers.bc8735934d-5919-4fa7-bdb1-8befa94690c2
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:25 librocksdbjni8850648580910683768.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:26 libgandiva_jni.soef27dee0-2b40-4055-a412-371eeee7ef14
-rw-r–r-- 1 dremio dremio 146K May 16 11:26 irhelpers.bcc513b672-3338-4efa-8b4b-8ad62b4ec8f3
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:26 librocksdbjni1128199500014421850.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:27 libgandiva_jni.so072f09b5-134a-4a8a-807d-d55d0ac3c9b2
-rw-r–r-- 1 dremio dremio 146K May 16 11:27 irhelpers.bc22883e51-f659-4d97-951e-bdce94e5b5b8
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:27 librocksdbjni4660007239191371976.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:28 libgandiva_jni.so51279231-e264-4ccd-8d08-1735104cd143
-rw-r–r-- 1 dremio dremio 146K May 16 11:28 irhelpers.bc17f48029-1feb-46f7-ac8a-0763a9ca0e5f
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:29 librocksdbjni3808416489901804549.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:29 libgandiva_jni.so0fc47bab-8ebc-4a7c-9fe9-9e6bb0110589
-rw-r–r-- 1 dremio dremio 146K May 16 11:29 irhelpers.bc76ccd155-39d1-4133-aeed-8de6d2c051ff
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:29 librocksdbjni3002667637748772840.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:30 libgandiva_jni.so64f06d6e-57a8-4a99-be35-927b6384a7ef
-rw-r–r-- 1 dremio dremio 146K May 16 11:30 irhelpers.bcf84b7ec3-a172-4331-abba-6867bbfa4235
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:31 librocksdbjni8036948367534165272.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:31 libgandiva_jni.so08062516-274b-47e8-ac24-58af5a83c354
-rw-r–r-- 1 dremio dremio 146K May 16 11:31 irhelpers.bc75bd17d3-85a1-4a15-8cb0-55c5c29d12a2
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:32 librocksdbjni3320584694528516618.so
-rw-r–r-- 1 dremio dremio 27M May 16 11:32 libgandiva_jni.so9faadb7b-896b-48fc-9076-fa0d73d24280
-rw-r–r-- 1 dremio dremio 0 May 16 11:32 libgandiva_jni.so96d2f14f-21a5-471d-8a6f-fbdcef55272a
drwxr-xr-x 2 dremio dremio 108 May 16 11:33 jna–1323653796
-rw-r–r-- 1 dremio dremio 0 May 16 11:33 librocksdbjni7547400547210792976.so
-rw-r–r-- 1 dremio dremio 0 May 16 11:33 librocksdbjni2354933948589969896.so
-rw-r–r-- 1 dremio dremio 0 May 16 11:34 librocksdbjni2888830322657183489.so
-rw-r–r-- 1 dremio dremio 0 May 16 11:34 librocksdbjni5673017548043232479.so
drwxr-xr-x 2 dremio dremio 20 May 16 11:34 hsperfdata_dremio

This suggests Dremio is trying to use the wrong jars. Can you do a recursive ls of your Dremio’s jars folder and post it here ? Thanks!
Also how did you install Dremio, rpm, tar ?

I installed via RPM.