Executor node not showing in admin panel after update to 3.0.1

Hi guys.
After updated Dremio to 3.0.1 version, my executor node is not been shown on admin panel, just the coordinator node. I’ve checked the /var/log/dremio/server.out and my executor instance was initialized correctly (it’s showing ‘Dremio Daemon Started as a slave’). Does anybody know what could happen?

When I check the jobs logs, I found this error:

Hi @Paulo_Vasconcellos

Click on Admin-Node activity then please click on jobs, choose internal jobs under UI-External Tools. You should see the below query, open the profile for that job and see if you see any error like “Failed to Create Spill directory”. Fix the error and make sure the below query runs and then node activity should show all the executor nodes

select
‘green’ as status,
nodes.hostname name,
nodes.ip_address ip,
nodes.fabric_port port,
cpu cpu,
memory memory
from
sys.nodes,
(select
hostname,
fabric_port,
sum(cast(cpuTime as float) / cores) cpu
from
sys.threads
group by
hostname,
fabric_port) cpu,
(select
hostname,
fabric_port,
direct_current * 100.0 / direct_max as memory
from
sys.memory) memory
where
nodes.hostname = cpu.hostname
and nodes.fabric_port = cpu.fabric_port
and nodes.hostname = memory.hostname
and nodes.fabric_port = memory.fabric_port
order by
name,
port

Thanks
@balaji.ramaswamy

Thank you for replying @balaji.ramaswamy

This is the error when I check the profile: VALIDATION ERROR: Unable to find the referenced field: [FileSize].

Just a fact: I followed this tutorial to upgrade Dremio, but this tutorial doesn’t say to upgrade Dremio on executor nodes. Is that right? Should I just upgrade on master and master coordinator?

Hi @Paulo_Vasconcellos

You do not have to run dremio-admin upgrade command on the executor nodes but you do have to do it on the master. Once the master is upgraded , you have to start any remaining coordinators and the executors from the new binary. When you log on to the Dremio UI and click Help-About Dremio, what version does it show?

Thanks
@balaji.ramaswamy

3.0.1-201811132128360291-804fe82

HI @Paulo_Vasconcellos

If it is possible can you please send me the profile for that failed job?

Thanks
@balaji.ramaswamy

Sure. Here it is.

2a1e8bae-8809-4a1a-93d9-431e3a79a83a.zip (8.7 KB)

Does anybody know how to solve this problem?

I’ve solved the problem by updating the Dremio nodes with the same version of the master.

1 Like