What is the memory config I should set for each node in the Dremio cluster

1

Hi there,

Right now, in our cluster, we leave the Memory setting as default
We un-touch all the following configs:
DREMIO_MAX_MEMORY_SIZE_MB
DREMIO_MAX_HEAP_MEMORY_SIZE_MB
DREMIO_MAX_DIRECT_MEMORY_SIZE_MB

Since the usage of Dremio is getting higher among our company, we’re occurring more and more memory issues, especially for Full GC

image
image1894×169 17.6 KB

5k Full GC events in 1 working day :frowning:

And it happens on all the nodes
image

So we want to tuning the setting for the memory
I was said that depends on the role, each node will need a specific setting for the memory
My cluster consists of:
1 x master: 32cpu, 96GB
1 x coordinator: 32cpu, 96GB
3 x executor: 32cpu, 372GB
3 x executor dedicated for reflection: 32cpu, 372GB

We’ve tried to pump the specs of each node really hard, but seems like the misconfiguration of the memory is the root cause here

Please help to advice

Thank you

2

@quangth2 What kind of deployment is this? Are you able to send ps -ef | grep dremio output from the nodes?