How can I set up cluster_Id for my dremio cluster

1

There is already a dremio cluster named ‘dremio’ now, I want to generate another cluster named ‘dremio2’. How can I do this?
if I can create a new cluster, how to set the path for Multiple Dremio Clusters Using a Single Zookeeper Cluster?

https://docs.dremio.com/advanced-administration/zookeeper.html

2

Hey there,

Would you mind elaborating on your question. What of the documentation isn’t clear? The documentation describes how to configure each Dremio cluster to point to an external zookeeper cluster. Are you asking how to configure your external zookeeper cluster?

Christy

3

hi, Thanks for your reply. I just want to know how to identified the clusterID. I need many dremio clusters, and each cluster have one master node. When I start dremio, the default clusterId is ‘dremio’ and I did not find somewhere to redefine it.

4

https://docs.dremio.com/advanced-administration/zookeeper.html#multiple-dremio-clusters-using-a-single-zookeeper-cluster
You are able to change this to your needs, for example -

zookeeper: “host:2181/new_cluster_id,host:2181/new_cluster_id,host:2181/new_cluster_id”

You are correct - “When no ZooKeeper path is specified, Dremio defaults to /dremio”

5

That’s OK! Thank you! The dremio should be installed, configured and started at each host, is that right?

6

what if the master node is not in the host list?

for example, master node is server1, and the conf file is just like follows, how server1 knows the cluster_id of itself is dremio2? :

services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: false
}
zookeeper: “server2:2181/dremio2,server3:2181/dremio2”

7

Can you talk a little about your environment and deployment?

Since you are using an external ZK, am I correct to assume you have a Hadoop cluster? If so, you can deploy the executors via our UI, so you will only need to install Dremio on 1 node that will be the master coordinator node.
For deployments without Hadoop, we would recommend using our internal built-in ZK, where then you would have the other nodes point to the master coordinator’s internal ZK.

8

hi,

when I set conf like:
zookeeper: “host1:5181/dremio2,host2:5181/dremio2,host3:5181/dremio2”

the log is:
DEBUG c.d.s.coordinator.zk.ZKClusterClient - Connect: host1:5181, zkRoot: dremio2,host2:5181/dremio2,host3:5181, clusterId: dremio

you can see that the clusterId is still dremio

9

I have a Hadoop cluster. And I can add a maprfs source of my cluster. But I did not find where I can deploy the executors in UI

10

Look under “Provisioning” tab on Admin