How to deploy executors in UI

Caused by: twill.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Path must not end with / character
106859 at twill.com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1015) ~[dremio-twill-shaded-2.0.1-201804132202000759-10b1de0-mapr.jar:2.0.1-201804132202000759-10b1de0-mapr]
106860 at twill.com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1001) ~[dremio-twill-shaded-2.0.1-201804132202000759-10b1de0-mapr.jar:2.0.1-201804132202000759-10b1de0-mapr]
106861 at org.apache.twill.zookeeper.ForwardingZKClientService.startAndWait(ForwardingZKClientService.java:51) ~[dremio-twill-shaded-2.0.1-201804132202000759-10b1de0-mapr.jar:2.0.1-201804132202000759-10b1de0-mapr]
106862 at org.apache.twill.yarn.YarnTwillRunnerService.startUp(YarnTwillRunnerService.java:350) ~[dremio-twill-shaded-2.0.1-201804132202000759-10b1de0-mapr.jar:2.0.1-201804132202000759-10b1de0-mapr]
106863 at org.apache.twill.yarn.YarnTwillRunnerService.access$000(YarnTwillRunnerService.java:112) ~[dremio-twill-shaded-2.0.1-201804132202000759-10b1de0-mapr.jar:2.0.1-201804132202000759-10b1de0-mapr]
106864 at org.apache.twill.yarn.YarnTwillRunnerService$3.startUp(YarnTwillRunnerService.java:171) ~[dremio-twill-shaded-2.0.1-201804132202000759-10b1de0-mapr.jar:2.0.1-201804132202000759-10b1de0-mapr]
106865 at twill.com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43) ~[dremio-twill-shaded-2.0.1-201804132202000759-10b1de0-mapr.jar:2.0.1-201804132202000759-10b1de0-mapr]
106866 … 1 common frames omitted
106867 Caused by: java.lang.IllegalArgumentException: Path must not end with / character
106868 at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:58) ~[zookeeper-3.4.5-mapr-1604.jar:3.4.5-mapr-1604–1]
106869 at org.apache.zookeeper.client.ConnectStringParser.(ConnectStringParser.java:57) ~[zookeeper-3.4.5-mapr-1604.jar:3.4.5-mapr-1604–1]
106870 at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:443) ~[zookeeper-3.4.5-mapr-1604.jar:3.4.5-mapr-1604–1]
106871 at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380) ~[zookeeper-3.4.5-mapr-1604.jar:3.4.5-mapr-1604–1]
106872 at org.apache.twill.internal.zookeeper.DefaultZKClientService$ServiceDelegate.createZooKeeper(DefaultZKClientService.java:537) ~[dremio-twill-shaded-2.0.1-201804132202000759-10b1de0-mapr.jar:2.0.1-201804132202000759-10b1de0-ma
106873 at org.apache.twill.internal.zookeeper.DefaultZKClientService$ServiceDelegate.doStart(DefaultZKClientService.java:447) ~[dremio-twill-shaded-2.0.1-201804132202000759-10b1de0-mapr.jar:2.0.1-201804132202000759-10b1de0-mapr]
106874 at twill.com.google.common.util.concurrent.AbstractService.start(AbstractService.java:170) ~[dremio-twill-shaded-2.0.1-201804132202000759-10b1de0-mapr.jar:2.0.1-201804132202000759-10b1de0-mapr]
106875 at org.apache.twill.internal.zookeeper.DefaultZKClientService.start(DefaultZKClientService.java:306) ~[dremio-twill-shaded-2.0.1-201804132202000759-10b1de0-mapr.jar:2.0.1-201804132202000759-10b1de0-mapr]
106876 at org.apache.twill.zookeeper.ForwardingZKClientService.start(ForwardingZKClientService.java:46) ~[dremio-twill-shaded-2.0.1-201804132202000759-10b1de0-mapr.jar:2.0.1-201804132202000759-10b1de0-mapr]
106877 … 6 common frames omitted

Just as a quick thought, have you set this in your dremio.conf files:

services.coordinator.master.embedded-zookeeper.enabled: false

This ensures Dremio uses the external zookeeper rather than the internal one. You’ll need to set this (and the zookeeper settings) for all nodes

yes, I already set this in dremio config file. I don’t understand you said that I need to set it for all node. I have only one config file which is on the master node. I don’t know where I can deploy other nodes. I only have one master node right now, I thought if I can add the Yarn provisioning, I will have some executors. Is that right?

Sorry, you’re right. Long day, I forgot you’d mentioned you deploy using Yarn.

Could you show content of your dremio.conf and dremio-env?

paths: {

the local path for dremio to store data.

local: “/var/lib/dremio”

the distributed path Dremio data including job results, downloads, uploads, etc

dist: “maprfs://data/chenlini/pdfs”
}
services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: false
}
zookeeper: “host1:5181,host2:5181,host3:5181/dremio2/”

I did not add anything to dremio-env

Please try

zookeeper: “host1:5181/dremio2,host2:5181/dremio2,host3:5181/dremio2”

I tried, if I use “zookeeper: “host1:5181/dremio2,host2:5181/dremio2,host3:5181/dremio2””

the log is:

connect: host1:5181 zkroot: dremio2,host2:5181/dremio2,host3:5181 clusterId: dremio

the clusterId is still dremio. Actually there is already a dremio cluster named “dremio” in our system, I need create another dremio cluster. If I config like that, I will connect with the old cluster.

I forgot to mention - you said you added - services.coordinator.master.embedded-zookeeper.enabled: false

However, you shared your dremio.conf and it is not listed there. Try

zookeeper: “host1:5181/dremio2,host2:5181/dremio2,host3:5181/dremio2”
services.coordinator.master.embedded-zookeeper.enabled: false

Sorry, I just check it and find I already added it. It still shows the same log.

Can you share your full complete dremio.conf please? Also FYI any change to our conf will require a service restart.

paths: {
local: “/var/lib/dremio”
dist: “maprfs://data/chenlini/pdfs”
}
services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: false
}
services.coordinator.master.embedded-zookeeper.enabled: false
zookeeper: “host1:5181/dremio2,host2:5181/dremio2,host3:5181/dremio2”

this is the full complete dremio.conf, I did not add any thing else.

now the error info is “no cluster identity found”.
Exception in thread “main” java.lang.NullPointerException: No Cluster Ident
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java
at com.dremio.dac.daemon.DremioDaemon.checkVersion(DremioDaemon.java:89
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:125)

This is likely due to failed start-up attempts. Can you wipe our metadata db? rm -rf /var/lib/dremio/*

when I execute “rm -rf /var/lib/dremio/*”
There is no “No Cluster Ident” error, but the log still shows:

DEBUG c.d.s.coordinator.zk.ZKClusterClient - Connect: host1:5181, zkRoot: dremio2,host2:5181/dremio2,host3:5181, clusterId: dremio

Hi, is there any idea about this problem? I just wan to generate a dremio cluster named ‘dremio2’. thanks.

Can you attach your full sever.log & server.out? Can you also check your external ZK logs to see if there are any errors?

There is no errors info, I can run dremio,and I can also add mapr source, but I can not create the dremio cluster. which means I can not add provisioning.

Per your other post here, it seems like the ZK issue is fixed (it now uses dremio2)
[main] DEBUG c.d.s.coordinator.zk.ZKClusterClient - Connect: host1:5181,host2:5181,host3:5181, zkRoot: , clusterId: dremio2

Actually, it’s not been fixed. I just changed the default cluster id in some of lib jar of dremio.