Install Dremio 4 errors

1

I am having dremio 3.3.2-201908142136370993-d60145d and would like to install version 4.
Both will be setup for some time until we stop and use 4.
Currently i am facing below error while starting

Dremio is exiting. Failure while starting services.
com.dremio.common.exceptions.UserException: Tried to access non-existent source [__jobResultsStore].
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:776)
at com.dremio.exec.catalog.CatalogServiceImpl.synchronize(CatalogServiceImpl.java:437)
at com.dremio.exec.catalog.CatalogServiceImpl.getPlugin(CatalogServiceImpl.java:887)
at com.dremio.exec.catalog.CatalogServiceImpl.getSource(CatalogServiceImpl.java:915)
at com.dremio.dac.daemon.DACDaemonModule.lambda$build$2(DACDaemonModule.java:434)
at com.dremio.service.jobs.LocalJobsService.start(LocalJobsService.java:280)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:184)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:190)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:104)

This error is after cleaning up data directory as configured in paths below

paths: {

the local path for dremio to store data.

local: “/opt/dremio4/data”

the distributed path Dremio data including job results, downloads, uploads, etc

dist: “hdfs://xxx.xxx.x.xx:8020/data/dremio4”
results: ${paths.local}/results
}

and my services config is
services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: false
coordinator.master.embedded-zookeeper.enabled: false
coordinator.web.port: 9049

coordinator.client-endpoint.port: 31019

coordinator.auto-upgrade: false
}

i use the cloudera zookeeper instance.

if i restart dremio after the above error , i am getting

Dremio is exiting. Failure while starting services.
com.dremio.common.exceptions.UserException: No Cluster Identity found
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:776)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:170)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:160)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:99)
Fri Sep 20 16:01:07 UTC 2019 Starting dremio on training-edgename01.xxxxxxxxx.com
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 254366
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 254366
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Unable ot come out of the circle. Can anyone help on this.

2

Did you run the upgrade command?

3

@mmohan

Did you take a backup before upgrading. This error usually means your db is out of sync. If you have a backup then restore backup go back to earlier version, make sure it comes up nice and clean, then follow upgrade steps

Upgrading Dremio

4

@doron, No i haven’t done that. Thought as the upgrade step is not needed as both of the versions were needed for us.

5

Is the DB residing in the paths:local which is mentioned in config. I saw a DB folder there while startup. I cleared the error no cluster identity found was gone after i cleared the contents of this folder. I removed the dist: “/path” and dremio 4 instance was able to start without errors. So the issue is with accessing the HDFS folder. i have changed the permission of that to 777 still not working. Anything else needs to be considered here?

Another thing is i am unable to start both the instances. Is it because at a time only one coordinator can exist in a host? I am unsing an external zookeeper which came with cloudera installation. And using tarball installation. Coordinator and executor nodes are same for both versions.

6

@mmohan

dist:// needs read/write/execute for user running the Dremio process. Also in HA configuration only one master-coordinator can be active. If this is not HA then we do not recommend 2 coordinators

7

i have removed the dist in my config and it worked. It was able to read fro hive so ignoring dist for now.