No Executors Available

1

I see this error. Previously on multiple new cluster invocation… I did not see any problem. Dremio is running on all servers. On Web UI the executors node activity is none.

018-08-11 07:56:43,731 [24916944-2cd6-a9bf-e0a6-78fd312b6200:foreman] INFO  c.d.e.p.s.h.c.ExecutionPlanCreator - User Error Occurred [ErrorId: d646622c-a5fe-4a04-9433-d247b9f67832]

com.dremio.common.exceptions.UserException: No executors currently available.
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.planner.sql.handlers.commands.ExecutionPlanCreator.getExecutionPlan(ExecutionPlanCreator.java:58) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:68) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:293) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
2018-08-11 07:56:43,732 [24916944-2cd6-a9bf-e0a6-78fd312b6200:foreman] INFO query.logger - {“queryId”:“24916944-2cd6-a9bf-e0a6-78fd312b6200”,“schema”:“sys”,“queryText”:“select\n ‘green’ as status,\n nodes.hostname name,\n nodes.ip_address ip,\n nodes.fabric_port port,\n cpu cpu,\n memory memory \nfrom\n sys.nodes,\n (select\n hostname,\n fabric_port,\n sum(cast(cpuTime as float) / cores) cpu \n from\n sys.threads \n group by\n hostname,\n fabric_port) cpu,\n (select\n hostname,\n fabric_port,\n direct_current * 100.0 / direct_max as memory \n from\n sys.memory) memory \nwhere\n nodes.hostname = cpu.hostname \n and nodes.fabric_port = cpu.fabric_port \n and nodes.hostname = memory.hostname \n and nodes.fabric_port = memory.fabric_port \norder by\n name,\n port”,“start”:1533974203540,“finish”:1533974203732,“outcome”:“FAILED”,“username”:“data_services_admin”,“commandDescription”:""}
2018-08-11 07:57:43,976 [24916908-3725-6c80-3183-33bbe726f900:foreman] WARN c.d.exec.planner.logical.RexToExpr - Converting exact decimal into approximate decimal. Should be fixed once decimal is implemented.
2018-08-11 07:57:43,981 [24916908-3725-6c80-3183-33bbe726f900:foreman] INFO c.d.e.p.s.h.c.ExecutionPlanCreator - User Error Occurred [ErrorId: 82a79010-1973-4d7e-ac8a-4747c30a17e4]
com.dremio.common.exceptions.UserException: No executors currently available.
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.planner.sql.handlers.commands.ExecutionPlanCreator.getExecutionPlan(ExecutionPlanCreator.java:58) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:68) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:293) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]

2

On executors, I see the following in the log, which is waiting…but, somehow, the coordinator is not seeing the executors.

2018-08-11 07:28:52,484 [main] INFO c.d.d.s.exec.MasterStatusListener - Waiting for master
2018-08-11 07:29:52,486 [main] INFO c.d.d.s.exec.MasterStatusListener - Waiting for master
2018-08-11 07:30:52,488 [main] INFO c.d.d.s.exec.MasterStatusListener - Waiting for master
2018-08-11 07:31:52,490 [main] INFO c.d.d.s.exec.MasterStatusListener - Waiting for master
2018-08-11 07:32:52,492 [main] INFO c.d.d.s.exec.MasterStatusListener - Waiting for master
2018-08-11 07:33:52,493 [main] INFO c.d.d.s.exec.MasterStatusListener - Waiting for master

3

A common reason we see Waiting for master is when there are multiple Dremio instances in 1 ZK path. Have you had another Dremio use the same ZK before? You can fix this by setting a new path - https://docs.dremio.com/advanced-administration/zookeeper.html#multiple-dremio-clusters-using-a-single-zookeeper-cluster

4

@anthony Thanks… it’s a glitch. I’ve been running single ZK cluster for single Dremio Cluster operation. This one was a glitch the ZK setup failed; but, fixed.