How dremio manage between heap and direct memory

1

For example, I have a 64GB instance.
I will set DREMIO_MAX_MEMORY_SIZE_MB to 60GB and leave the rest for OS use.

  1. How Dremio split these 60 GB between heap and direct?
  2. Is it safe to max this setting to match instance memory capacity (i.e 64 GB)?
2

Hi @chafidz

Let me start by answering #2,
#2. Is it safe to max this setting to match instance memory capacity (i.e 64 GB)? It is not safe and not recommended, the reason being the OS needs some memory for performing internal operations. When doing those if it finds there i no memory available it invokes the oom-killer (Google for "oom-killer). The oom-killer wakes up and looks for the process that is consuming the most memory. In your case it would be Dremio and will get killed, you would see something like the below in your /var/log/messages

Feb 1 18:07:02 kernel: output.rb:140 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

It is recommended that you leave at least 1-2 GB to the OS. 60 GBB to Dremio souns like a good start. Also if there are other applications running on the same box, you have make sure the OS does not swap due to lack of memory

#1. How Dremio split these 60 GB between heap and direct?

If you give DREMIO_MAX_MEMORY_SIZE_MB = 60 GB, by default 4 GB would be heap (might change if you have more for Dremio). You can run the below command and see how much has gone to heap and how much to direct

ps -ef | grep dremio

Look for below 2 parameters (-Xmx is heap and -XX: is Direct)

-Xmx4096m -XX:MaxDirectMemorySize=8192m

On your coordinator if you have large datasets sometimes the default value of 4 GB may not be enough. Going too high on heap can also cause Full GC pauses

Kindly let us know if you have any further questions

Thanks
@balaji.ramaswamy

1 Like
3

Thanks for the explanation. This help us a lot!