How to set dremio max memory?

1

Hi,dremio friend:
when I config the value for dremio in file /opt/dremio/conf/dremio-env like this:
DREMIO_LOG_DIR=/var/log/dremio
DREMIO_PID_DIR=/var/run/dremio
DREMIO_MAX_HEAP_MEMORY_SIZE_MB=8192
DREMIO_MAX_MEMORY_SIZE_MB=16384
#DREMIO_MAX_HEAP_MEMORY_SIZE_MB=8192
DREMIO_MAX_DIRECT_MEMORY_SIZE_MB=16384

but the dremio start log like this

Dremio Daemon Started as master
Tue Dec 25 09:25:11 CST 2018 Starting dremio on gpmaster
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 160637
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 131072
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
KVStore version is 3.0.0-201810262305460004-5c90d75
max memory size (kbytes, -m) unlimited

this describe my dremio’s max memory size is unlimited

2

These messages describes what kind of limits your operating systems (Linux) may set on the process, but the settings in dremio-env instructs the Java virtual machine how much memory to be used in practice.

3

Hi, Mr. laurent . thank your reply my problem . how to set the dremio max memory ?

4

Hi @bjbhjsj

In /opt/dremio/conf/dremio-env,

Either set DREMIO_MAX_MEMORY_SIZE_MB=16384
(OR)
DREMIO_MAX_HEAP_MEMORY_SIZE_MB=8192
DREMIO_MAX_DIRECT_MEMORY_SIZE_MB=16384

When you give DREMIO_MAX_MEMORY_SIZE_MB, the Heap gets 4GB and the rest for direct. On the coordinator you do not require much of direct memory while on the executor you can keep the heap at the minimum 4GB

If you are running a single node Dremio then just set DREMIO_MAX_MEMORY_SIZE_MB. Do not go very close to the OS memory as the OS needs some for itself.

Thanks
@balaji.ramaswamy

5

thanks @balaji.ramaswamy
I had one physical machine(256G memory) which Four virtual machine every for 40G OS memory, I give every dremio for 16384M, run reflection query from single table ( 670,000,000 records) base on greenplum(postgresql cluster)