Salut. Resource consumption question
Due to certain requirements, we use dremio as master and executor (at the same time), without dedicating separate executors. Dremio running on a machine with 128 GB of RAM
There are several questions about resource allocation.
Memory.
According to the documentation, available memory is specified using parameters (parameter and how I understand it)
DREMIO_MAX_MEMORY_SIZE_MB - maximum memory used by dremio
DREMIO_MAX_HEAP_MEMORY_SIZE_MB - Heap memory, used to run dremio itself. Coordinators use it to a greater extent.
DREMIO_MAX_DIRECT_MEMORY_SIZE_MB - direct memory, used to execute requests. Executors use it to a greater extent.
At the same time, in the section best_practices for executor it is stated that "
Note: Even if you have a machine with 64GB of RAM, only 16GB is used by default. To change this setting, modify the DREMIO_MAX_DIRECT_MEMORY_SIZE_MB property in the dremio-env file and restart the executor node(s).
Do I understand correctly that in master + executor mode, having 128 GB of RAM, I need to directly set DREMIO_MAX_HEAP_MEMORY_SIZE_MB and DREMIO_MAX_DIRECT_MEMORY_SIZE_MB? Or is it enough to specify one of them and specify DREMIO_MAX_MEMORY_SIZE_MB, after which the entire balance will automatically go to another?
Are there any recommendations for memory allocation with our configuration? For example, 32 GB for heap and the rest for Direct?
I understand that you need to leave resources to the system
CPU
Will the CPU be maxed out even without load? Or will the load increase in fact, and if we don’t run complex queries (with calculations in dremio), then we won’t see a high load?
thx