@balaji.ramaswamy coordinator node have 16 GB and each executor node have 32GB
@balaji.ramaswamy
Coordinator
dremio 3328 1 99 Jan19 ? 2-21:02:24 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler -Djava.library.path=/opt/dremio/lib -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/dremio/server-%t.gc -Ddremio.log.path=/var/log/dremio -Ddremio.plugins.path=/opt/dremio/plugins -Xmx4096m -XX:MaxDirectMemorySize=8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/dremio -Dio.netty.maxDirectMemory=0 -Dio.netty.tryReflectionSetAccessible=true -DMAPR_IMPALA_RA_THROTTLE -DMAPR_MAX_RA_STREAMS=400 -Xloggc:/var/log/dremio/server-%t.gc -XX:+UseG1GC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2000 -XX:GCLogFileSize=50M -XX:+StartAttachListener -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC -cp /opt/dremio/conf:/opt/dremio/jars/:/opt/dremio/jars/ext/:/opt/dremio/jars/3rdparty/:/var/dremio_efs/thirdparty/:/usr/lib/jvm/java-1.8.0-openjdk/lib/tools.jar com.dremio.dac.daemon.DremioDaemon
ed2-user 4284 3613 0 13:14 pts/0 00:00:00 grep --color=auto dremio
dremio 6672 3328 0 Jan19 ? 00:14:08 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.342.b07-1.amzn2.0.1.x86_64/jre/bin/java -Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler -Djava.library.path=/opt
dremio/lib -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/dremio/preview/server.gc -Ddremio.log.path=/var/log/dremio/preview -Ddremio.plugins.path=/opt/dremio/plugins -Xmx4096m -XX:MaxDirectMemorySize=8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/dremio/preview -Dio.netty.maxDirectMemory=0 -Dio.netty.tryReflectionSetAccessible=true -DMAPR_IMPALA_RA_THROTTLE -DMAPR_MAX_RA_STREAMS=400 -Xloggc:/var/log/dremi
/server-%t.gc -XX:+UseG1GC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2000 -XX:GCLogFileSize=50M -XX:+StartAttachListener -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC -cp /etc/dremio/preview:/opt/dremio/jars/:/opt/dremio/jars/ext/:/opt/dremio/jars/3rdparty/:/var/dremio_efs/thirdparty/ com.dremio.dac.daemon.DremioDaemon
Executor
dremio 3287 1 12 Jan17 ? 11:55:46 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.amzn2.0.2.x86_64/jre/bin/java -Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler -Djava.library.path=/opt
dremio/lib -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/dremio/server.gc -Ddremio.log.path=/var/log/dremio -Ddremio.plugins.path=/opt/dremio/plugins -Xmx4096m -XX:MaxDirectMemorySize=24534m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/dremio -Dio.netty.maxDirectMemory=0 -Dio.netty.tryReflectionSetAccessible=true -DMAPR_IMPALA_RA_THROTTLE -DMAPR_MAX_RA_STREAMS=400 -XX:+UseG1GC -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC -cp /opt/dremio/conf:/opt/dremio/jars/:/opt/dremio/jars/ext/:/opt/dremio/jars/3rdparty/:/var/dremio_efs/thirdparty/ com.dremio.dac.daemon.DremioDaemon
ed2-user 26247 26171 0 13:22 pts/0 00:00:00 grep --color=auto dremio
@Diego Few things
- After you have increased your direct memory to 24 GB, have you run the query in Job ID#
223529ef-69d5-5f6e-742d-cee7981b1c00again?I yes, send me the profile, if not, please run it and see what happens, if it fails, send the profile again - You are on a very archived version of Dremio 3.x, please upgrade to ASAP as at some point, we are not able to support you just because the code has significantly changed and we cannot even troubleshoot this anymore
- Memory requirements depends on # and type of queries you have, we can only advice on why a query ran OOM but if you are looking for answers like
- “How many executor nodes do I need”?
- What will be the direct/heap memory required for each node
These answers are not the same for every cluster and depends on the workload and you would have to work with an architect , again the version you are on does not have many things that are on 21.x, for example, Dremio 3.x does not have the concept of Engines where different workloads can be clubbed into it’s own pool of executors, reflection matching algorithms cover much more matching patterns, coordinator and Executor heap memory usage are far more optimized, if you are reading Parquet then the read performance of the Parquet files are much more optimized, I can keep going on but this is something, you need to take a step back and think about
Kindly let us know, if you have any other questions
@balaji.ramaswamy I created a new node and assigned it the preview queue. I removed the executor role from the master node. Apparently this is the solution to my problem.
Thanks for the update, yes, it is not recommended to run the master also as an executor. Good catch