Dremio ui query ui very slow

1

I have used Azure AKS Deployment , migrated dremio to 3.2 2-3 months back,before that ui is opening very fast

but after that i see gui is too slow ,but if i kill pods and restart pod using helm for few days dremio gui working fast but after that again its slow till next restart

here is the output for ps -ef | grep dremio on master node

image
image.png1919×159 14.6 KB

Note: i am using only one node for Master ,executer

I have also clean metadata using https://docs.dremio.com/advanced-administration/metadata-cleanup.html but still same issues

image
image.png1827×871 30.5 KB

2

Our next major release will include some memory leak fixes that may be causing what you are seeing. If the release doesn’t help we can investigate further.

3

Thanks, I am looking forward for next release

4

Dremio gui again too slow althorugh i have updated dremio 4.0.2,can find any Error logs

image
image1884×933 96.8 KB

5

@Vikash_Singh

Is the UI slow to load objects or you get this error?

6

Ui are very slow suddenly but if I restart pods all looks good it’s a issue from past 6 month can’t able to solve

7

@Vikash_Singh,

Can you collect jstack when UI is slow? We will look into that and get back.
Here’s an example jstack script.

for i in seq -w 3 1 300
do
jstack -l 22788 > ThreadDump$i.txt
sleep 1
done

Please replace 22788 with the actual pid of your Dremio coordinator. And the user to run this script needs to be the same user who starts Dremio service.

8

sorry to say that i cant able to run above Script as i have kubernate pods Container ,
i am getting below Error

dremio@dremio-master-0:/tmp$ jstack -l -F 1
Attaching to process ID 1, please wait…
ERROR: ptrace(PTRACE_ATTACH, …) failed for 1: Operation not permitted
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can’t attach to the process: ptrace(PTRACE_ATTACH, …) failed for 1: Operation not permitted
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: Can’t attach to the process: ptrace(PTRACE_ATTACH, …) failed for 1: Operation not permitted
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:163)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:278)
at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:671)
at sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611)
at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:337)
at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:304)
at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
at sun.tools.jstack.JStack.main(JStack.java:106)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: Can’t attach to the process: ptrace(PTRACE_ATTACH, …) failed for 1: Operation not permitted
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$100(LinuxDebuggerLocal.java:62)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1AttachTask.doit(LinuxDebuggerLocal.java:269)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:138)

9

@Ye_Li @balaji.ramaswamy Please find attached file for Jvm dump running on co-ordinator log

ThreadDump.zip (8.6 KB)