Memory leak in Dremio

I see that for some reason free memory on the node with Dremio seems to be leaking (I use single node on-premise setup); see screenshot attached.

After Dremio restart the memory gets free but then continues to leak.
In server.gc there are tons of GC (Allocation Failure) messages.

Is there any way I can fix it?

Hi @stifstyleAre you using G1GC for garbage collection. If not please implement G1GC and observe the failures.
What is the Dremio version your using ?



What is the memory setting on the Dremio node. Send us the server.gc* files and we can review


Thank you very much for your advice; errors in server.gc disappeared after implementing G1GC but memory seems to continue leaking.
I use Dremio version 4.3.1-202005202256080999-5dcfb82a

Uploaded server.gc.* log files here

I run Dremio on AWS i3.2xlarge instance.

The only settings I specified are the following:
DREMIO_JAVA_SERVER_EXTRA_OPTS="-XX:+UseG1GC -XX:G1HeapRegionSize=32M -XX:MaxGCPauseMillis=500 -XX:InitiatingHeapOccupancyPercent=25"
and maximum number of file descriptors.


The total RAM on the box on a “AWS i3.2xlarge instance.” is 60 GB
Your direct memory is 56 GB
Your heap memory is 4 GB

You have no space left for the OS, you are probably getting killed by “oom-killer”

Please check “/var/log/messages” for “oom-killer” around the time Dremio is going down

Leave at least 4-8 GB for OS

If this is coordinator, increase heap to 12 GB and decrease 16 GB from direct (make it 40 GB)
If this is executor, increase heap to 8 GB, decrease 12 GB from direct (make it 44 GB)


Thanks, will try to leave more memory for OS. But the thing is, Dremio is not getting killed because I monitor free memory on server and restart Dremio before OOM. What bothers me is that free memory is constantly decreasing until I restart Dremio.

By the way, I use single node setup of Dremio; is that a bad practice?


Yes, running Dremio on a single node is not recommended for production workloads