Consumed cpu and ram resources

Salut. Resource consumption question

Due to certain requirements, we use dremio as master and executor (at the same time), without dedicating separate executors. Dremio running on a machine with 128 GB of RAM

There are several questions about resource allocation.

According to the documentation, available memory is specified using parameters (parameter and how I understand it)
DREMIO_MAX_MEMORY_SIZE_MB - maximum memory used by dremio
DREMIO_MAX_HEAP_MEMORY_SIZE_MB - Heap memory, used to run dremio itself. Coordinators use it to a greater extent.
DREMIO_MAX_DIRECT_MEMORY_SIZE_MB - direct memory, used to execute requests. Executors use it to a greater extent.

At the same time, in the section best_practices for executor it is stated that "

Note: Even if you have a machine with 64GB of RAM, only 16GB is used by default. To change this setting, modify the DREMIO_MAX_DIRECT_MEMORY_SIZE_MB property in the dremio-env file and restart the executor node(s).

Do I understand correctly that in master + executor mode, having 128 GB of RAM, I need to directly set DREMIO_MAX_HEAP_MEMORY_SIZE_MB and DREMIO_MAX_DIRECT_MEMORY_SIZE_MB? Or is it enough to specify one of them and specify DREMIO_MAX_MEMORY_SIZE_MB, after which the entire balance will automatically go to another?

Are there any recommendations for memory allocation with our configuration? For example, 32 GB for heap and the rest for Direct?
I understand that you need to leave resources to the system

Will the CPU be maxed out even without load? Or will the load increase in fact, and if we don’t run complex queries (with calculations in dremio), then we won’t see a high load?


@Arol Running C+E on a single node is generally not recommended but in you case you can so Heap as 16 GB, 8 GB leave it to OS and then 104 GB to DIRECT

Do free -g and make sure total is 128 GB, if it shows something like 124 then leave 4 GB to 6 GB to OS, 16 GB to heap and remaining to DIRECT

1 Like

Thank you. Yes, we understand that it is better to separate the master and the coordinator, but for now we have to do it this way.

Last questions:

Will it be enough for us to specify DREMIO_MAX_DIRECT_MEMORY_SIZE_MB 16GB and DREMIO_MAX_MEMORY_SIZE_MB 120GB, after which 104 will automatically go to HEAP? Or is it necessary to allocate DIRECT and HEAP ?

Is there an answer for CPU load?