How optimize memory use on executor

1

Hi,

We are using dremio with one coordinator and one executor about a month. The results we’re getting are great, but recently I did some stressful queries to test executor capabilities and received a “IllegalStateException: Memory was leaked by query. Memory leaked: (66560)”. Ok, this was the objective, but I noticed, by using jconsole to track jmx metrics how dremio doc suggest, that use of memory do not pass 1.5Gb (heap and non-heap).

My executor setup is a aws c5.2xlarge that have 16Gb of memory.

My dremio-env is with DREMIO_MAX_MEMORY_SIZE_MB=16384

So my question is, are there any setting I forgot? Because, I’m getting a memory error but there is a lot of memory available on server.

thanks,

2

Dremio uses two kinds of memory: heap and direct. With 16GB of memory configured, you would get 14gb/2gb heap/direct memory for a standalone coordinator, 4gb/12gb of heap/direct memory for an executor.

As for this message, it only pertains to direct memory (not heap), and you should be able to identify which query(ies) cause this leak, as Dremio creates a separate allocator for each query. If you can share profiles of the query to help us identify the cause of the leak, this would be very helpful.