What is the benefit of having multiple coordinators

#1

Hi,

What is the role and benefit of having multiple coordinator nodes in a dremio cluster?

#2

Hi @Sneha_Krishnaswamy

If you are having concurrent queries and want to distribute the planning time then having multiple coordinators might help. As a first step, are you having multiple queries at the same time and experiencing high planning times?

Thanks
@balaji.ramaswamy

#3

I’m just exploring dremio as of now, so right now we don’t have concurrent queries.
So, essentially all coordinator nodes can perform query planning and not just the master node, is it?

#4

That’s right @Sneha_Krishnaswamy,

coordinator can also be master or just coordinator. But only one coordinator can be the master (talks to rocksDB) and you should not configure two coordinators to be both masters

services: {

  coordinator: {
    enabled: true,

    # Auto-upgrade Dremio at startup if needed
    auto-upgrade: false,

    master: {
      enabled: true,
      # configure an embedded ZooKeeper server on the same node as master
      embedded-zookeeper: {
        enabled: true,
        port: 2181,
        path: ${paths.local}/zk
      }
    },
#5

Hi @Sneha_Krishnaswamy,

Just to confirm that when I said both coordinators cannot be masters is in a non HA mode when both coordinators are active and planning queries. Of course you can have a HA setup where we can have 2 masters in 2 coordinators but only one master coordinator will be active and can plan queries

Thanks
@balaji.ramaswamy

#6

understood… thank you for the detailed explanation.

#7

One last question though, could you direct me to some documentation on how to create an HA setup?

#8

https://docs.dremio.com/advanced-administration/high-availability.html