Some times, one query spend a long time in planning fase, in some others, it’s instantly
Does somebody have an idea about the problem and how to solve it?
I attached some files pointing out the problem, one faster and another slower.
5adc0184-6bb1-4257-bd39-e328399693d9.zip (11,3,KB)
a1dc3577-dde1-466b-82f3-8f2d170fae99.zip (11,5,KB)
Do you have a sample mongo document from the above collection you can provide? In case the data is not sensitive?
vwTempoDeCalculo-sampleDocument.zip (1,2,KB)
My dremio runs in a machine with 32GB RAM and 8 CPU cores
this machine contains three dockers containers:
one with dremio (DREMIO_MAX_MEMORY_SIZE_MB = 16384)
one with mongodb (it sees 16GB of RAM)
one with a service in nodejs
It could be possible that when the planning time is high there are Full GC’s happening. I also see one another issue. You have distributed the total RAM of 32 GB between MongoDB and Dremio leaving nothing for the OS. Can you please GC logs when the query spends lot of time in planning?
Thanks
@balaji.ramaswamy
Where do I found this GC logs?
I am monitoring the machine with the top command. The Dremio never uses more than 2GB of RAM
On the Dremio container, do a “ps -ef | grep dremio”, look for “-Xloggc” in the output and you will see the path to server.gc. If you do not see that flag then it will be part of server.log
Thanks
@balaji.ramaswamy
This is a new query with spend 9min in planning and the gc logs
servergc.zip (241,2,KB)
19m35s - 97c3e927-ac9e-43cd-8d9d-0be21937854a.zip (9,7,KB)
I reviewed the server settings.
in fact the dremio has 20GB and the Mongodb 8GB and not 16GB for each
Hi @oluis
Can you please try to enable “accelerator.simplified_match” and retry query?
I have enabled this key.
But there are still queries that continue to take a long time (over 10 min) in “planning time”.
Is there anything relating to machine settings I can do?
Or can I limit the planning time and make the query cancel if it takes too long?
Kindly send me the query profile after you enabled the parameter and still planning takes a lot of time
low-1df3c566-1634-44dd-8716-2089fd656907.zip (14,6,KB)
low-accelerated-621a443e-f78e-47c3-ac5a-4cfd1ebfc338.zip (8,7,KB)
I have two keys ennabled:
