Dremio so far has been a tour of cryptic error messages for me. The latest error I get is “IllegalArgumentException: No executors available to assign work.” I’ve imported 4 datasets from Excel spreadsheets with roughly 10K rows and 3-4 columns in each. I have tried letting Dremio sit for 30m. I checked there are no running jobs. I’ve restarted several times. Such a promising tool but unusable because it gets so easily wedged with no obvious way to recover.
Can you check in the admin page how many executor nodes you have ?
1 node. I’m just running this locally on OSX.
Could you check “Admin” panel in Dremio UI. See screenshot with some highlights. Do you have your local machine showing there or none?
So when it is showing: “IllegalArgumentException: No executors available to assign work” - is there is anything in the log? Both server.log and server.out
Tailing both server.log and server.out, there is no activity while I’m not doing anything. Once I try to execute any query, the following gets logged:
==> server.log <==
2017-07-25 11:09:10,036 [268884ca-6ecc-fa6f-1648-3efacf8be700:foreman] INFO query.logger - {“queryId”:“268884ca-6ecc-fa6f-1648-3efacf8be700”,“schema”:“”,“queryText”:“SELECT *\nFROM "@nhartner"."pk-nps"”,“start”:1501002549999,“finish”:1501002550036,“outcome”:“FAILED”,“username”:“nhartner”,“commandDescription”:“”}
2017-07-25 11:09:10,040 [268884ca-6ecc-fa6f-1648-3efacf8be700:foreman] ERROR c.d.exec.work.foreman.AttemptManager - IllegalArgumentException: No executors available to assign work.
com.dremio.common.exceptions.UserException: IllegalArgumentException: No executors available to assign work.
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:622) ~[dremio-common-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]
at com.dremio.exec.work.foreman.AttemptManager$AttemptResult.close(AttemptManager.java:556) [dremio-sabot-kernel-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]
at com.dremio.exec.work.foreman.AttemptManager.moveToState(AttemptManager.java:660) [dremio-sabot-kernel-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]
at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:333) [dremio-sabot-kernel-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: com.dremio.exec.work.foreman.ForemanException: Unexpected exception during fragment initialization: No executors available to assign work.
… 4 common frames omitted
Caused by: java.lang.IllegalArgumentException: No executors available to assign work.
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) ~[guava-18.0.jar:na]
at com.dremio.exec.store.schedule.AssignmentCreator2.getMappings(AssignmentCreator2.java:53) ~[dremio-sabot-kernel-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]
at com.dremio.exec.planner.fragment.Wrapper.assignEndpoints(Wrapper.java:116) ~[dremio-sabot-kernel-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]
at com.dremio.exec.planner.fragment.HardAffinityFragmentParallelizer.parallelizeFragment(HardAffinityFragmentParallelizer.java:133) ~[dremio-sabot-kernel-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]
at com.dremio.exec.planner.fragment.SimpleParallelizer.parallelizeFragment(SimpleParallelizer.java:299) ~[dremio-sabot-kernel-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]
at com.dremio.exec.planner.fragment.SimpleParallelizer.getFragmentsHelper(SimpleParallelizer.java:215) ~[dremio-sabot-kernel-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]
at com.dremio.exec.planner.fragment.SimpleParallelizer.getFragments(SimpleParallelizer.java:167) ~[dremio-sabot-kernel-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]
at com.dremio.exec.planner.sql.handlers.commands.ExecutionPlanCreator.getExecutionPlan(ExecutionPlanCreator.java:85) ~[dremio-sabot-kernel-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]
at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:68) ~[dremio-sabot-kernel-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]
at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:285) [dremio-sabot-kernel-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]
… 3 common frames omitted
On startup, I am seeing the following WARN message every time:
017-07-25 11:19:06,290 [main] INFO c.d.s.s.LocalSchedulerService - SchedulerService is up
2017-07-25 11:19:06,291 [main] INFO c.d.service.jobs.LocalJobsService - Starting JobsService
2017-07-25 11:19:06,359 [main] INFO c.d.service.jobs.LocalJobsService - JobsService is up
2017-07-25 11:19:06,450 [scheduler-0] WARN c.d.s.s.LocalSchedulerService - Execution of task com.dremio.service.jobs.LocalJobsService$CleanupTask@5b6d4714 failed
java.lang.NullPointerException: null
at com.dremio.service.jobs.LocalJobsService$CleanupTask.run(LocalJobsService.java:1453) ~[dremio-services-jobs-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]
at com.dremio.service.scheduler.LocalSchedulerService$CancellableTask.run(LocalSchedulerService.java:124) ~[dremio-services-scheduler-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Could you check what you have in dremio.conf for following section:
master: {
# the name of the master server. If this node matches the name, it starts the master service
name:
port:
}
as well as /etc/hosts (and/or ifconfig)
If you have that ip that is listed on your screenshot mapped to some domain/host. May be try to comment it out or use it for “master” in dremio.conf
In /Applications/Dremio.app/Contents/Java/dremio/conf/dremio.conf I had the following:
master: {
name: localhost,
port: 45678
}
Changing localhost to 127.0.0.1 and restarting seemed to fix the issue. The IP it was resolving to before in the nodes lists was the correct IP assigned to me by DHCP so not sure why that wasn’t working. But loopback IP works so I will just go with that.
Thanks!
I am glad it got resolved with the workaround. We will be looking into this issue.