Memory Issue on planning phase

1

Hello,
When i submit a query that join 2 table, got the error:
dremio query was cancelled because it exceeded the memory planning.

message-error
message-error1280×45 19 KB

After that I increase the memory on executors, from 96GB > 102 > 110, but still get this error. At the end, I have to setup 128GB on each executor, and then it work fine. I checked on profile file, nodes memory consume just around 15GB. Can you support me on this case?

2

Hi @nghiabq What version of Dremio is this?

3

Hello @balaji.ramaswamy, Dremio version is 19.0 enterprise, using Yarn for executors

4

@nghiabq Can you please enable planner.memory.aggressive and then retry the job?

5

hi @balaji.ramaswamy,
what is the meaning of planner.memory.aggressive key? I encounter the same issue, and I don’t have any mem limit in the queue workload management

6

@quangth2 Is your query also failing with out of memory? Can you please send the job profile?

7

hi @balaji.ramaswamy

for more precise, the reflection creation job is failing because the memory exceeding the limit set by admin. but for the queue workload on reflection, I did not set any limit at all, on both mem and time as well

image
image1182×353 45.1 KB

image
image421×743 25.6 KB

Please take a look at the job profile
b4d1c2c3-01be-4c40-bd1d-3771b4a82989.zip (3.6 MB)

8

@quangth2 What is max direct memory on the executor? Can you please send us ps -ef | grep dremio output from an executor?

9

It’s really high @balaji.ramaswamy. our cluster is running on a very big machine

-Xmx51200m -Xms51200m -XX:MaxDirectMemorySize=321536m

10

@quangth2 Let me check on this and get back to you

11

@balaji.ramaswamy for more detail, we have 6 executor nodes with the same specs & configuration

12

@quangth2 I think I just replied on your ticket today. We have opened an internal bug with Engineering on this, will keep you posted