Planning time differ in same query

1

Some times, one query spend a long time in planning fase, in some others, it’s instantly

Does somebody have an idea about the problem and how to solve it?

I attached some files pointing out the problem, one faster and another slower.

5adc0184-6bb1-4257-bd39-e328399693d9.zip (11,3,KB)
a1dc3577-dde1-466b-82f3-8f2d170fae99.zip (11,5,KB)

2

@oluis

Do you have a sample mongo document from the above collection you can provide? In case the data is not sensitive?

3

vwTempoDeCalculo-sampleDocument.zip (1,2,KB)

4

My dremio runs in a machine with 32GB RAM and 8 CPU cores
this machine contains three dockers containers:
one with dremio (DREMIO_MAX_MEMORY_SIZE_MB = 16384)
one with mongodb (it sees 16GB of RAM)
one with a service in nodejs

5

@oluis

It could be possible that when the planning time is high there are Full GC’s happening. I also see one another issue. You have distributed the total RAM of 32 GB between MongoDB and Dremio leaving nothing for the OS. Can you please GC logs when the query spends lot of time in planning?

Thanks
@balaji.ramaswamy

6

Where do I found this GC logs?

I am monitoring the machine with the top command. The Dremio never uses more than 2GB of RAM

7

@oluis

On the Dremio container, do a “ps -ef | grep dremio”, look for “-Xloggc” in the output and you will see the path to server.gc. If you do not see that flag then it will be part of server.log

Thanks
@balaji.ramaswamy

1 Like
8

This is a new query with spend 9min in planning and the gc logs

servergc.zip (241,2,KB)
19m35s - 97c3e927-ac9e-43cd-8d9d-0be21937854a.zip (9,7,KB)

9

I reviewed the server settings.
in fact the dremio has 20GB and the Mongodb 8GB and not 16GB for each

10

Hi @oluis

Can you please try to enable “accelerator.simplified_match” and retry query?

11

I have enabled this key.
But there are still queries that continue to take a long time (over 10 min) in “planning time”.
Is there anything relating to machine settings I can do?
Or can I limit the planning time and make the query cancel if it takes too long?

12

@oluis

Kindly send me the query profile after you enabled the parameter and still planning takes a lot of time

13

low-1df3c566-1634-44dd-8716-2089fd656907.zip (14,6,KB)
low-accelerated-621a443e-f78e-47c3-ac5a-4cfd1ebfc338.zip (8,7,KB)

I have two keys ennabled:
image