Can I use server status API for executors running in kubernetes?

1

Can I use the endpoint /apiv2/server_status on the executor nodes? Which port should I use when using this API on the executors?

I am running this in Kubernetes, and can check the master node’s health using the above, but am unable to figure out if I can also monitor executor health with this API.

2

@recurse Have you got the solution for this?

3

@unni @recurse Would not work for an executor as 9047 is only on the coordinator. Is the idea to monitor executors?

4

@balaji.ramaswamy Yes, the purpose is to monitor executors.
Once or twice we got a situation when the pods were in the RUNNING state but no requests were received were processed by the pod.

In another case we got the error
The source ["__accelerator"] is currently unavailable, Unknown protocol.

We restarted the executor to resolve the issue.

5

@unni That could be a different monitoring action, executor is up but the distributed storage did not attach to this executor and restart fixed it, maybe you can monitor the log during the executor startup and make sure the plugins starting with “__” show healthy

6

@balaji.ramaswamy We are using local storage. I will monitor logs for this.

7

@unni Try this

open dremio.conf on your executors and add below line

services.web-admin.port: 9091

Restart the executor, hit the below end point and if it loads at once then executor is healthy

http://<host_or_ip>:9091/metrics

1 Like