What is the benefit of having multiple coordinators

1

Hi,

What is the role and benefit of having multiple coordinator nodes in a dremio cluster?

2

Hi @Sneha_Krishnaswamy

If you are having concurrent queries and want to distribute the planning time then having multiple coordinators might help. As a first step, are you having multiple queries at the same time and experiencing high planning times?

Thanks
@balaji.ramaswamy

3

I’m just exploring dremio as of now, so right now we don’t have concurrent queries.
So, essentially all coordinator nodes can perform query planning and not just the master node, is it?

4

That’s right @Sneha_Krishnaswamy,

coordinator can also be master or just coordinator. But only one coordinator can be the master (talks to rocksDB) and you should not configure two coordinators to be both masters

services: {

  coordinator: {
    enabled: true,

    # Auto-upgrade Dremio at startup if needed
    auto-upgrade: false,

    master: {
      enabled: true,
      # configure an embedded ZooKeeper server on the same node as master
      embedded-zookeeper: {
        enabled: true,
        port: 2181,
        path: ${paths.local}/zk
      }
    },
5

Hi @Sneha_Krishnaswamy,

Just to confirm that when I said both coordinators cannot be masters is in a non HA mode when both coordinators are active and planning queries. Of course you can have a HA setup where we can have 2 masters in 2 coordinators but only one master coordinator will be active and can plan queries

Thanks
@balaji.ramaswamy

6

understood… thank you for the detailed explanation.

7

One last question though, could you direct me to some documentation on how to create an HA setup?

8

https://docs.dremio.com/advanced-administration/high-availability.html

9

@balaji.ramaswamy
In what situations that JDBC and ODBC connections will be running on the secondary coordinator? For example heavy loading of master coordinator.

10

@Ming Dremio at a given time can plan not more than # of cpu’s - 1, so if you have 32 cores on the coordinator, Dremio can plan 31 in parallel. Say the dash board fires 155 queries in parallel, then the first 31 get planned (command pool time), while the remaining 124 queries are in PENDING state. Say each query in the 1st set of 31 queries take 1s to plan, the last batch of the 31 queries will have a 4s wait time before they can get planned. So depending on your concurrency and where jobs come from like JDBC or ODBC or REST, you can add a secondary coordinator only for planning and currently only for JDBC and ODBC queries

11

@balaji.ramaswamy
I tried to write a python coding for testing. Our coordinator has 8 cores. I simulated 10,000 concurrent sessions to connect the coordinator. I do not find any messages in server.log from secondary coordinator. How can I do to know the secondary coordinator is used?

12

@Ming I assume the Python code is ODBC (or JDBC), the job profile should tell you the name of the coordinator