Zookeeper configuration, please help

1

Hi everyone

I am making a cluster, but i am having trouble visualizing the executor nodes. I can only see the coordinator node in the administration part:

image
image1287×280 22.7 KB

The coordinator node settings is as follows:

paths: {
  # the local path for dremio to store data.
  local: "/var/lib/dremio"

  # the distributed path Dremio data including job results, downloads, uploads, etc
  #dist: "pdfs://"${paths.local}"/pdfs"
}

services: {
  coordinator.enabled: true,
  coordinator.master.enabled: true,
  executor.enabled: false,
  #coordinator.master.embedded-zookeeper.port: 31010
  #coordinator.master.embedded-zookeeper.enabled: false
}

zookeeper: "dremio-coord-01:2181"

And de executor node configuration:

paths: {
  # the local path for dremio to store data.
  local: "/var/lib/dremio"

  # the distributed path Dremio data including job results, downloads, uploads, etc
  #dist: "pdfs://"${paths.local}"/pdfs"
}

services: {
  coordinator.enabled: false,
  coordinator.master.enabled: false,
  executor.enabled: true,
  #coordinator.master.embedded-zookeeper.port: 31010
  #coordinator.master.embedded-zookeeper.enabled: false
}

zookeeper: "dremio-coord-01:2181"

I appreciate any help, thanks in advance!

2

if yuo are using external zookeeper you must set coordinator.master.embedded-zookeeper.enabled: false
also all nodes must be can resolve all host to ips, (you can use custon nameserver o edit hosts file)

3

Thanks for your answer! I still don’t understand what they mean when they talk about an external zookeeper…

I created four virtual machines, they all ping each other. I installed a dremio in each one of them and now I am doing the configurations, however when I enter through the client I don’t see the exec nodes.

when you tell me to edit the host file, do you have an example used with dremio that I can see?

Thanks a lot!

4

@gaston_guerra

  • Dremio recommend up a cluster using yarn or other provision tool, not recommend manual config cluster, but thi is possible. Please follow this instructiont and upload your log files from master and nodes
  • Dremio uses zookeeper for coordination between nodes and master so this mean that all nodes must be resolvable throgh hostname, por example if you have 4 machines
  • dremio-coord-01
  • dremio-exec-01
  • dremio-exec-02
  • dremio-exec-03
    en each odes you need to define hostnames and ips in /etc/hosts, so you can test
    from dremio-coord-01

ping dremio-exec-01

and all beetween nodes.

ALSO you must open ports in firewall https://docs.dremio.com/deployment/system-requirements.html#network

please follow ths instructions and upload logs.

5

Thanks a lot!!! The problem was that port 45678 was not open.
I really appreciate the help, have an amazing week!!!