Urgent: Dremio will not start after upgrade to 3.2.0

#1

even though I have used the dremio-admin upgrade command - which completed successfully, dremio will not start and I’m seeing the following error in the server.out file:

Dremio is exiting. Failure while starting services.
java.lang.IllegalStateException: KVStore has an older version (3.1.11-201904261857420193-c674472) than the server (3.2.0-201905102005330382-0598733), please run the upgrade tool first
at com.google.common.base.Preconditions.checkState(Preconditions.java:721)
at com.dremio.dac.daemon.DremioDaemon$AutoUpgrade.ensureUpgradeSupported(DremioDaemon.java:67)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:178)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:160)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:97)

and the following in server.log:

019-05-15 14:52:24,119 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
java.lang.IllegalStateException: KVStore has an older version (3.1.11-201904261857420193-c674472) than the server (3.2.0-201905102005330382-0598733), please run the upgrade tool first
at com.google.common.base.Preconditions.checkState(Preconditions.java:721) ~[guava-20.0.jar:na]
at com.dremio.dac.daemon.DremioDaemon$AutoUpgrade.ensureUpgradeSupported(DremioDaemon.java:67) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:178) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:160) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:97) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]

#2

@summersmd, are you absolutely sure your are pointing Dremio to the correct db? What do you have for paths.db in dremio.conf

#3

I’m sure I’m pointing to the correct db. It’s the same db we’ve been using and the configuration has not changed.

paths: {

the local path for dremio to store data.

local: “/app/dremio/data”

the distributed path Dremio data including job results, downloads, uploads, etc

dist: “pdfs://”${paths.local}"/pdfs"

location for catalog database (if master node)

db: ${paths.local}/db,

spilling: [${paths.local}/spill]

storage area for the accelerator cache.

accelerator: ${paths.dist}/accelerator

staging area for json and csv ui downloads

downloads: ${paths.dist}/downloads

stores uploaded data associated with user home directories

uploads: ${paths.dist}/uploads

stores data associated with the job results cache.

results: ${paths.dist}/results

shared scratch space for creation of tables.

scratch: ${paths.dist}/scratch
}

#4

I reran ./dremio-admin upgrade as the dremio user account (instead of root) and now I’m getting a new error in server.out:

Dremio is exiting. Failure while starting services.
java.lang.NoSuchMethodError: org.eclipse.jetty.http.pathmap.PathMappings.put(Lorg/eclipse/jetty/http/pathmap/PathSpec;Ljava/lang/Object;)Z
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1421)
at org.eclipse.jetty.servlet.ServletHandler.doStart(ServletHandler.java:165)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:123)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:913)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:365)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.server.Server.start(Server.java:415)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:382)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at com.dremio.dac.server.WebServer.start(WebServer.java:257)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102)

#5

and the following in server.log:

2019-05-15 15:11:17,755 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
java.lang.NoSuchMethodError: org.eclipse.jetty.http.pathmap.PathMappings.put(Lorg/eclipse/jetty/http/pathmap/PathSpec;Ljava/lang/Object;)Z
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1421) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.servlet.ServletHandler.doStart(ServletHandler.java:165) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:123) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:913) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:365) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.Server.start(Server.java:415) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.server.Server.doStart(Server.java:382) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at com.dremio.dac.server.WebServer.start(WebServer.java:257) ~[dremio-dac-backend-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137) ~[dremio-common-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74) ~[dremio-common-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33) ~[dremio-common-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183) ~[dremio-dac-backend-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189) ~[dremio-dac-backend-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]

#6

@summersmd

What version of JDK are you using? Also who owns the Dremio files? Is it Dremio or root?

Thanks
@balaji.ramaswamy

#7

@balaji.ramaswamy dremio owns the files. Using java 1.8.0_211.

#8

Hi @summersmd

Is this a stand alone node? If not, did we bring down the executors and upgrade the software there too?

Thanks
@balaji.ramaswamy

#9

This is a standalone node.

#10

Ok to recap

JDK 1.8 running Dremio 3.1
Upgraded software to 3.2 also ran dremio-admin upgrade?
Now Dremio is not starting.

If this is the case kindly send us the server.log

Thanks
@balaji.ramaswamy

#11

Confirmed:
JDK 1.8 running Dremio 3.1
Upgraded software to 3.2 also ran dremio-admin upgrade?
Now Dremio is not starting.

Attached is the server.log file.server.log.zip (588.7 KB)

#12

I see the below. Can you please do a df -h from the mount rocksDB is on?

2019-05-15 19:29:21,974 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
java.lang.ExceptionInInitializerError: null
at org.rocksdb.Options.(Options.java:25) ~[rocksdbjni-5.14.2.jar:na]
at com.dremio.datastore.ByteStoreManager.start(ByteStoreManager.java:201) ~[dremio-services-datastore-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.datastore.CoreStoreProviderImpl.start(CoreStoreProviderImpl.java:195) ~[dremio-services-datastore-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.datastore.LocalKVStoreProvider.start(LocalKVStoreProvider.java:157) ~[dremio-services-datastore-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:158) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:97) ~[dremio-dac-daemon-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]
Caused by: java.lang.RuntimeException: Unable to load the RocksDB shared libraryjava.io.IOException: No space left on device
at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:67) ~[rocksdbjni-5.14.2.jar:na]
at org.rocksdb.RocksDB.(RocksDB.java:35) ~[rocksdbjni-5.14.2.jar:na]
… 6 common frames omitted

#13

How is the location of the rocksdb configured? I see our /tmp directory is 100% full and it looks like the rocksdb is saved there, but I’m not sure why it would be saved there. It looks like I need to change the location for rocksdb to something other than /tmp.

#14

Here is the documentation

https://docs.dremio.com/advanced-administration/dremio-conf.html

paths: {

  # the local path for dremio to store data.
  local: ${DREMIO_HOME}"/data"
#15

In our dremio.conf file we have:

paths: {
local: “/app/dremio/data”

which is the path we’ve used for all of our previous dremio installs and we’ve never had this issue before.

for some reason, /tmp is being used.

#16

Can you please send df -h /tmp and ls -ltrh /tmp outputs?

#17

I think I have fixed the issue with /tmp running out of space, however now I’m now getting a new error message. I’m growing concerned because I have a proof of concept demo on Monday. The new error message I’m seeing is:

Dremio is exiting. Failure while starting services.
java.lang.NoSuchMethodError: org.eclipse.jetty.http.pathmap.PathMappings.put(Lorg/eclipse/jetty/http/pathmap/PathSpec;Ljava/lang/Object;)Z
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1421)
at org.eclipse.jetty.servlet.ServletHandler.doStart(ServletHandler.java:165)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:123)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:913)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:365)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.server.Server.start(Server.java:415)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:382)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at com.dremio.dac.server.WebServer.start(WebServer.java:257)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102)

#18

I stand corrected, the longer I leave dremio running, the more files it writes to /tmp.

df -h /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-tmp 2.0G 2.0G 0 100% /tmp


ls -lrth /tmp | grep dremio
drwxr-xr-x 2 dremio dremio 6 May 7 01:38 dremio-support
-rw-r–r-- 1 dremio dremio 146K May 15 16:38 irhelpers.bc30552ffd-1b09-4457-89fe-e98ae0fb3abb
-rw-r–r-- 1 dremio dremio 7.9M May 15 16:42 librocksdbjni2474945149489956128.so
-rw-r–r-- 1 dremio dremio 36M May 15 16:43 libgandiva_jni.so1819c659-5827-447e-b304-42ce168ef62c
-rw-r–r-- 1 dremio dremio 146K May 15 16:43 irhelpers.bc88666329-edd4-47a9-9f5d-ac61bdb14e61
-rw-r–r-- 1 dremio dremio 7.9M May 15 16:43 librocksdbjni4768622427663942288.so
-rw-r–r-- 1 dremio dremio 36M May 15 16:44 libgandiva_jni.so3fc95d50-fdd4-4a62-8390-251b4a007531
-rw-r–r-- 1 dremio dremio 146K May 15 16:44 irhelpers.bcceb9a66f-f7c9-4c95-80af-a17ae8566153
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:43 librocksdbjni5417623524275247120.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:45 libgandiva_jni.soc7b195ec-4a18-4c30-ae60-eab0617b7897
-rw-r–r-- 1 dremio dremio 146K May 16 10:45 irhelpers.bc2bab94f2-1ac8-45e7-866d-151bdf6373c9
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:46 librocksdbjni8384188306582098660.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:46 libgandiva_jni.so5a81702f-800c-456c-b0e6-da0ca2cee2d8
-rw-r–r-- 1 dremio dremio 146K May 16 10:46 irhelpers.bc7f23d312-91b5-4a61-92ed-134c33faa156
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:47 librocksdbjni4062246913097261205.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:48 libgandiva_jni.so8293c76e-f36f-4c49-98d9-6b1d53cd90fb
-rw-r–r-- 1 dremio dremio 146K May 16 10:48 irhelpers.bcec4f6f66-f9d5-40dc-8887-7738a30dc5fa
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:48 librocksdbjni6280709598633111714.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:49 libgandiva_jni.so18b92d7c-e1fa-4661-b606-459e50e63940
-rw-r–r-- 1 dremio dremio 146K May 16 10:49 irhelpers.bc5de7f42d-0d92-4c7d-9be0-86f3c079d4ac
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:49 librocksdbjni3667222257910104154.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:50 libgandiva_jni.so03f8c0aa-a4ff-4b0a-98d0-42afffd07c4c
-rw-r–r-- 1 dremio dremio 146K May 16 10:50 irhelpers.bcfaf206b8-2225-4c17-a530-90f70e2824d9
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:50 librocksdbjni228048964999189158.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:51 libgandiva_jni.so0297a3ef-5508-4bf4-ad7b-a18598cf8102
-rw-r–r-- 1 dremio dremio 146K May 16 10:51 irhelpers.bcde8ef139-06cf-4e7e-8d80-28dbbf6bd470
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:52 librocksdbjni7997952340478346314.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:52 libgandiva_jni.so27e55edb-498e-4617-a105-da7e38b3cfe3
-rw-r–r-- 1 dremio dremio 146K May 16 10:52 irhelpers.bc063085a0-a73b-4fa0-a4b1-0399526211e5
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:53 librocksdbjni3154744454342548913.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:53 libgandiva_jni.so9f84671d-1c87-4f40-bc5e-94187f779041
-rw-r–r-- 1 dremio dremio 146K May 16 10:53 irhelpers.bcb4545f10-ccc8-409d-9503-e63eae7f51ec
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:54 librocksdbjni3331418003058889195.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:54 libgandiva_jni.so5a35ccd7-9b1d-49f1-a54d-70f8efe2d87a
-rw-r–r-- 1 dremio dremio 146K May 16 10:54 irhelpers.bc5ed0ca94-e8c0-4758-99a7-5e59348b6742
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:55 librocksdbjni3780990704380170840.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:55 libgandiva_jni.so020eee57-a951-4cb9-9967-55cb46a34c03
-rw-r–r-- 1 dremio dremio 146K May 16 10:55 irhelpers.bcee3b5181-a029-4833-9a08-a49685845815
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:56 librocksdbjni8770384686193740154.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:56 libgandiva_jni.sof8ae024c-a582-4ce6-aa01-93046bc3dbc4
-rw-r–r-- 1 dremio dremio 146K May 16 10:56 irhelpers.bc12f2b8a7-10f2-4a6e-bc6d-11f00c4bb569
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:57 librocksdbjni2868343319686299212.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:58 libgandiva_jni.soe82390f7-0e26-4ad2-ad30-c08fd7f84baa
-rw-r–r-- 1 dremio dremio 146K May 16 10:58 irhelpers.bceee56ef3-e867-4b9d-bc3e-bf17ee51d4e0
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:58 librocksdbjni5430611665186786377.so
-rw-r–r-- 1 dremio dremio 36M May 16 10:59 libgandiva_jni.so194879e8-eb3f-484a-9392-a3abcbe3f94c
-rw-r–r-- 1 dremio dremio 146K May 16 10:59 irhelpers.bc22c67afc-ff55-4f6d-a484-2156e8bcff84
-rw-r–r-- 1 dremio dremio 7.9M May 16 10:59 librocksdbjni16987508368763531.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:00 libgandiva_jni.so8fd88c80-8948-48b1-9925-59c3ba97f432
-rw-r–r-- 1 dremio dremio 146K May 16 11:00 irhelpers.bcbb147a92-6d85-4682-a1c6-3ffea6ac3e6c
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:00 librocksdbjni8495180989958949497.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:01 libgandiva_jni.sofc09a72c-7052-4848-967c-ecdb7e597b07
-rw-r–r-- 1 dremio dremio 146K May 16 11:01 irhelpers.bca33b70c7-c4b8-4233-b131-4af095f30d5d
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:01 librocksdbjni2111038461817832352.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:02 libgandiva_jni.so05a21594-a1fb-419b-bfc7-4e7d455c91c9
-rw-r–r-- 1 dremio dremio 146K May 16 11:02 irhelpers.bc6354810a-1de8-480f-86ca-524ee93594f4
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:02 librocksdbjni4794141259125900654.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:03 libgandiva_jni.so3910938e-ba3c-4269-9ec5-46564568e1d5
-rw-r–r-- 1 dremio dremio 146K May 16 11:03 irhelpers.bce860641b-a6ad-49e3-940c-949a63c3b577
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:04 librocksdbjni6008862212927600634.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:04 libgandiva_jni.so1e37c37d-51d8-4357-9d7d-c78ca2a61936
-rw-r–r-- 1 dremio dremio 146K May 16 11:04 irhelpers.bc9e806f79-54f6-4bce-ae3f-f6dc830eacfc
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:05 librocksdbjni5438135808316540718.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:05 libgandiva_jni.so92b62f55-0580-4831-8e39-397a40e08848
-rw-r–r-- 1 dremio dremio 146K May 16 11:05 irhelpers.bcb3ba39e6-21e1-46b4-84a5-3d1a00f7e00d
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:06 librocksdbjni7790380156454323906.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:07 libgandiva_jni.sof664fc57-4269-4510-869a-376bd7baa5cd
-rw-r–r-- 1 dremio dremio 146K May 16 11:07 irhelpers.bcdc725a56-dc67-4529-af4e-fc76a1e76b87
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:07 librocksdbjni7635437086162451889.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:08 libgandiva_jni.so6c7cb961-608a-46d8-83d9-0e94c8c5b433
-rw-r–r-- 1 dremio dremio 146K May 16 11:08 irhelpers.bcf11d9569-352b-4d1d-a719-5b236c1bfb99
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:08 librocksdbjni6518991588091194938.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:09 libgandiva_jni.sofb81071d-4920-4a62-8b00-5507ed7db359
-rw-r–r-- 1 dremio dremio 146K May 16 11:09 irhelpers.bc9c3500e2-a299-448d-b000-9c1a53f0b4ee
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:09 librocksdbjni5389221458865006443.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:10 libgandiva_jni.so4599fbac-32a7-4eff-b833-1c4e59bdc3a3
-rw-r–r-- 1 dremio dremio 146K May 16 11:10 irhelpers.bcec2dea2b-e165-4bd0-a8d9-163ce71736e6
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:11 librocksdbjni97702939113759583.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:11 libgandiva_jni.soe593f0cb-b1d9-40f3-b6ab-65265d936eca
-rw-r–r-- 1 dremio dremio 146K May 16 11:11 irhelpers.bcb0f3fa6e-a6bd-4ca7-be2d-b5f489586869
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:12 librocksdbjni1096290942753668021.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:12 libgandiva_jni.soded70803-a06b-4e78-bd22-3f41a84dc88c
-rw-r–r-- 1 dremio dremio 146K May 16 11:12 irhelpers.bc207726a7-b32b-49a2-8ecd-d40231c87b46
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:13 librocksdbjni5369465178013541165.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:13 libgandiva_jni.so4b92956d-e86e-4bbd-8105-395a0876fb85
-rw-r–r-- 1 dremio dremio 146K May 16 11:13 irhelpers.bc76125676-b8f8-489a-b119-66ef767ba1a4
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:14 librocksdbjni915796185479156281.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:14 libgandiva_jni.soc1ae666a-69df-4056-967f-d260e6b34a12
-rw-r–r-- 1 dremio dremio 146K May 16 11:14 irhelpers.bc8cf0d620-324d-451d-afbc-f483e2157386
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:15 librocksdbjni100666521919505226.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:15 libgandiva_jni.so173963f2-7cac-4213-b10b-3b46687a4b58
-rw-r–r-- 1 dremio dremio 146K May 16 11:15 irhelpers.bc736c3d0f-c392-4d43-90ce-259b91662b2a
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:16 librocksdbjni6884246809784225427.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:16 libgandiva_jni.so92af4ba8-56b5-419f-8432-8bf32d025149
-rw-r–r-- 1 dremio dremio 146K May 16 11:16 irhelpers.bc6eb51d99-522b-4cbc-bd35-e08404c6861c
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:17 librocksdbjni4443737537568495709.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:17 libgandiva_jni.so5405510e-c62f-4448-8e47-11c3ec97b35d
-rw-r–r-- 1 dremio dremio 146K May 16 11:17 irhelpers.bc9d9cd87a-5ed8-4828-9b79-cc625485e6ba
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:18 librocksdbjni531952699970304132.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:18 libgandiva_jni.so9741607f-817b-4672-862b-afb9b6e9d188
-rw-r–r-- 1 dremio dremio 146K May 16 11:18 irhelpers.bcb9dedf33-f116-42fe-ae64-2c63d1216d33
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:19 librocksdbjni1208636884817099782.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:20 libgandiva_jni.so8b9af1e1-36e8-4b73-a8a1-3baf0329fe09
-rw-r–r-- 1 dremio dremio 146K May 16 11:20 irhelpers.bca7bfeb1c-e3a1-484a-9540-ca92a93de909
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:20 librocksdbjni2430962552333391106.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:21 libgandiva_jni.sob4f42290-dc46-4a86-b7b5-424ebe122a24
-rw-r–r-- 1 dremio dremio 146K May 16 11:21 irhelpers.bcd49a9f78-9bd1-4739-96a3-ab7df22ce150
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:21 librocksdbjni1362561622445855994.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:22 libgandiva_jni.so532a9ada-713a-4e5f-a446-8cef95a02e84
-rw-r–r-- 1 dremio dremio 146K May 16 11:22 irhelpers.bc98952b35-dee6-438f-88ad-416d0f40b1de
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:22 librocksdbjni4562727618918815357.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:23 libgandiva_jni.so6b67f25d-8d5c-4301-bd46-10079890279b
-rw-r–r-- 1 dremio dremio 146K May 16 11:23 irhelpers.bc5a877d8d-1029-412b-ad77-7506b1c7bf33
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:23 librocksdbjni6179365421258594994.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:24 libgandiva_jni.so38d3c3f6-4262-48a4-a15c-5c2f185e5385
-rw-r–r-- 1 dremio dremio 146K May 16 11:24 irhelpers.bccaa8fda4-ee1d-42c9-b62a-d6c1266d9964
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:24 librocksdbjni1775497409614260618.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:25 libgandiva_jni.so20ddb292-20a8-4d88-a3ea-45cf2624948c
-rw-r–r-- 1 dremio dremio 146K May 16 11:25 irhelpers.bc8735934d-5919-4fa7-bdb1-8befa94690c2
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:25 librocksdbjni8850648580910683768.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:26 libgandiva_jni.soef27dee0-2b40-4055-a412-371eeee7ef14
-rw-r–r-- 1 dremio dremio 146K May 16 11:26 irhelpers.bcc513b672-3338-4efa-8b4b-8ad62b4ec8f3
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:26 librocksdbjni1128199500014421850.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:27 libgandiva_jni.so072f09b5-134a-4a8a-807d-d55d0ac3c9b2
-rw-r–r-- 1 dremio dremio 146K May 16 11:27 irhelpers.bc22883e51-f659-4d97-951e-bdce94e5b5b8
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:27 librocksdbjni4660007239191371976.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:28 libgandiva_jni.so51279231-e264-4ccd-8d08-1735104cd143
-rw-r–r-- 1 dremio dremio 146K May 16 11:28 irhelpers.bc17f48029-1feb-46f7-ac8a-0763a9ca0e5f
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:29 librocksdbjni3808416489901804549.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:29 libgandiva_jni.so0fc47bab-8ebc-4a7c-9fe9-9e6bb0110589
-rw-r–r-- 1 dremio dremio 146K May 16 11:29 irhelpers.bc76ccd155-39d1-4133-aeed-8de6d2c051ff
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:29 librocksdbjni3002667637748772840.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:30 libgandiva_jni.so64f06d6e-57a8-4a99-be35-927b6384a7ef
-rw-r–r-- 1 dremio dremio 146K May 16 11:30 irhelpers.bcf84b7ec3-a172-4331-abba-6867bbfa4235
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:31 librocksdbjni8036948367534165272.so
-rw-r–r-- 1 dremio dremio 36M May 16 11:31 libgandiva_jni.so08062516-274b-47e8-ac24-58af5a83c354
-rw-r–r-- 1 dremio dremio 146K May 16 11:31 irhelpers.bc75bd17d3-85a1-4a15-8cb0-55c5c29d12a2
-rw-r–r-- 1 dremio dremio 7.9M May 16 11:32 librocksdbjni3320584694528516618.so
-rw-r–r-- 1 dremio dremio 27M May 16 11:32 libgandiva_jni.so9faadb7b-896b-48fc-9076-fa0d73d24280
-rw-r–r-- 1 dremio dremio 0 May 16 11:32 libgandiva_jni.so96d2f14f-21a5-471d-8a6f-fbdcef55272a
drwxr-xr-x 2 dremio dremio 108 May 16 11:33 jna–1323653796
-rw-r–r-- 1 dremio dremio 0 May 16 11:33 librocksdbjni7547400547210792976.so
-rw-r–r-- 1 dremio dremio 0 May 16 11:33 librocksdbjni2354933948589969896.so
-rw-r–r-- 1 dremio dremio 0 May 16 11:34 librocksdbjni2888830322657183489.so
-rw-r–r-- 1 dremio dremio 0 May 16 11:34 librocksdbjni5673017548043232479.so
drwxr-xr-x 2 dremio dremio 20 May 16 11:34 hsperfdata_dremio

#19

This suggests Dremio is trying to use the wrong jars. Can you do a recursive ls of your Dremio’s jars folder and post it here ? Thanks!
Also how did you install Dremio, rpm, tar ?

#20

I installed via RPM.

ls ls.zip (15.7 KB)
-lR results attached here.