Dremio HA on K8s - web port in chart v1 vs v2

1

Hi

I wonder why the coordinator’s in helm chart v2 does not expose web port 9047.
This slightly breaks the HA of Dremio in k8s, and I’m not sure if there is a reason not to loadbalance traffic to all coordinator nodes (master and coordinators).

Chart v1 was exposing port 9047 to a coordinator node

while chart v2 does not

With chart v2, if the dremio-master is unavailable, dremio api is unavailable (even if a coordinator is working correctly in the cluster).

The docs states

Dremio’s web application can be made highly available by leveraging more than one coordinator node and a reverse proxy/load balancer.
All web clients connect to a the load balancer rather than directly connecting to an individual coordinator node. The load balancer distributes these connections to the current active master node.
https:// docs.dremio. com/current/get-started/cluster-deployments/customizing-configuration/dremio-conf/high-availability-config/ [sorry, only 2 links in the post]

It is not clear to me, how LB (the nginx in the example) could distinguish current master node out of all available coordinator nodes, as they all expose port 9047…

I did extend chart v2 dremio-coordinator.yaml with

   ports:

    - containerPort: 9047
      name: web

and it seems to work well, web and api remain available even if the dremio-master pod is turned of (given the coordinator is present in the cluster).

Is there any potential issue that could happen to the dremio cluster?

2

To give more insights, I have mainly workload that uses REST API calls, no ODBC/JDBC… not sure how this relatest to that quote

Configure multiple secondary coordinator nodes to improve concurrency and distribute query planning for ODBC and JDBC client requests to your deployment. To distribute queries across your deployment, Dremio recommends that you also configure ZooKeeper.
Secondary coordinator nodes run ODBC and JDBC queries only from clients that connect to the secondary coordinator either directly or with a load balancer.

3

@jmuszynski By default UI port is 9047, if master goes down it should come up on another pod automatically. HA is different from scale out and this is very different between K8s and the Cloud version On K8’s the master is SPOF as when it goes down the application is not accessible. If scale out goes down in software which only fails queries running on that scale out. In Cloud, coordinator is masterless and hence it is not a SPOF