Tasks die at service start

1

Hello,

I have a single node installation of a Dremio community server (v25.0.5) on a local machine.
I have installed it following the RPM guide. The data are stored on a filesystem.

Althought the Dremio console starts, I’m not able to do any query since they keep running forever. A simple SELECT 1 does not finish neither in COMPLETED or FAILED status. Upon restarting the service the query obviously fails.

At the start of the service I’ve noticed the following errors:

thread errors 1
thread errors 1888×932 117 KB

Basically it seems that 16 threads are started and they all die.

System information:
Linux RHEL 9.2, 8 CPU, 16 GB RAM, 50 GB disk.

The only configurations that I’ve chanegd are the following:

dremio-env:

DREMIO_JAVA_EXTRA="-Djava.io.tmpdir=/opt/dremio/tmp -Dcdjd.io.netty.tryReflectionSetAccessible=true"

I changed the tmp dir for privileges issues on the machine with the folder /tmp (there is a noexec policy)

dremio.conf:

paths: {
  # the local path for dremio to store data.
  # local: "/var/lib/dremio"

  # the distributed path Dremio data including job results, downloads, uploads, etc
  dist: "file:///mnt/dremio/nas"
}

services: {
  coordinator.enabled: true,
  coordinator.master.enabled: true,
  executor.enabled: true,
  flight.use_session_service: true,
  coordinator.web.ssl.enabled: true
}
2

Is there any particular path that must have executable grants ?
I changed /tmp but I see that there are also /var/lib or /var/run

3

@ancelot182 There is a similar bug that was fixed in 25.0 but you are already in 25.0.5. Let me check if there is something else going on

What happens if you remove -Dcdjd.io.netty.tryReflectionSetAccessible=true

At this point it does not look like a permission error

Do you have the full log file?

4

Can you also provide your dremio.conf?

5

Hi,

thanks for the reply here found the server.log

server.zip (95.0 KB)

Note that there are multiple restart since I was trying things.
The last one is at 13:09:29,753. After a bit you will find the logs about the threads:

2024-10-01 13:09:45,865 [e0] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e0 done
2024-10-01 13:09:45,865 [e9] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e9 done
2024-10-01 13:09:45,865 [e15] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e15 done
2024-10-01 13:09:45,866 [e2] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e2 done
2024-10-01 13:09:45,866 [e12] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e12 done
2024-10-01 13:09:45,866 [e7] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e7 done
2024-10-01 13:09:45,866 [e13] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e13 done
2024-10-01 13:09:45,866 [e10] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e10 done
2024-10-01 13:09:45,866 [e5] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e5 done
2024-10-01 13:09:45,866 [e11] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e11 done
2024-10-01 13:09:45,866 [e6] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e6 done
2024-10-01 13:09:45,865 [e8] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e8 done
2024-10-01 13:09:45,871 [e5] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e5,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
2024-10-01 13:09:45,872 [e13] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e13,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
2024-10-01 13:09:45,866 [e3] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e3 done
2024-10-01 13:09:45,866 [e4] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e4 done
2024-10-01 13:09:45,866 [e14] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e14 done
2024-10-01 13:09:45,866 [e0] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e0,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
2024-10-01 13:09:45,866 [e1] INFO  c.d.s.task.slicing.SlicingTaskPool - Deprecated slicing thread e1 done
2024-10-01 13:09:45,867 [e9] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e9,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
2024-10-01 13:09:45,869 [e2] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e2,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
2024-10-01 13:09:45,869 [e7] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e7,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
2024-10-01 13:09:45,871 [e12] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e12,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
2024-10-01 13:09:45,876 [e4] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e4,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
2024-10-01 13:09:45,872 [e6] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e6,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
2024-10-01 13:09:45,871 [e10] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e10,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
2024-10-01 13:09:45,872 [e11] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e11,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
2024-10-01 13:09:45,873 [e8] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e8,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
2024-10-01 13:09:45,876 [e1] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e1,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
2024-10-01 13:09:45,876 [e3] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e3,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
2024-10-01 13:09:45,876 [e14] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e14,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
2024-10-01 13:09:45,878 [e15] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[e15,5,main] died
java.lang.UnsupportedOperationException: null
	at net.openhft.affinity.impl.NullAffinity.getThreadId(NullAffinity.java:57)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:174)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:201)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)

For the dremio.conf is the basic one I posted in the OP.

Another note: I’ve tried to downgrade from 25.0.5 to 24.3.2 and I’ve the same problem.

6

@ancelot182 It looks like a mis-configuration as you have not defined a distributed storage, can you please send your dremio-env and dremio.conf removing any passwords

If a K8’s deployment, tar gz the helm charts and send it over,