Query was cancelled because it exceeded the memory limits set by the administrator

21

@balaji.ramaswamy coordinator node have 16 GB and each executor node have 32GB

22

@Diego Are you able to send ps -ef | grep dremio output from the coordinator and any one executor?

23

@balaji.ramaswamy
Coordinator
dremio 3328 1 99 Jan19 ? 2-21:02:24 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler -Djava.library.path=/opt/dremio/lib -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/dremio/server-%t.gc -Ddremio.log.path=/var/log/dremio -Ddremio.plugins.path=/opt/dremio/plugins -Xmx4096m -XX:MaxDirectMemorySize=8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/dremio -Dio.netty.maxDirectMemory=0 -Dio.netty.tryReflectionSetAccessible=true -DMAPR_IMPALA_RA_THROTTLE -DMAPR_MAX_RA_STREAMS=400 -Xloggc:/var/log/dremio/server-%t.gc -XX:+UseG1GC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2000 -XX:GCLogFileSize=50M -XX:+StartAttachListener -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC -cp /opt/dremio/conf:/opt/dremio/jars/:/opt/dremio/jars/ext/:/opt/dremio/jars/3rdparty/:/var/dremio_efs/thirdparty/:/usr/lib/jvm/java-1.8.0-openjdk/lib/tools.jar com.dremio.dac.daemon.DremioDaemon
ed2-user 4284 3613 0 13:14 pts/0 00:00:00 grep --color=auto dremio
dremio 6672 3328 0 Jan19 ? 00:14:08 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.342.b07-1.amzn2.0.1.x86_64/jre/bin/java -Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler -Djava.library.path=/opt
dremio/lib -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/dremio/preview/server.gc -Ddremio.log.path=/var/log/dremio/preview -Ddremio.plugins.path=/opt/dremio/plugins -Xmx4096m -XX:MaxDirectMemorySize=8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/dremio/preview -Dio.netty.maxDirectMemory=0 -Dio.netty.tryReflectionSetAccessible=true -DMAPR_IMPALA_RA_THROTTLE -DMAPR_MAX_RA_STREAMS=400 -Xloggc:/var/log/dremi
/server-%t.gc -XX:+UseG1GC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2000 -XX:GCLogFileSize=50M -XX:+StartAttachListener -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC -cp /etc/dremio/preview:/opt/dremio/jars/:/opt/dremio/jars/ext/:/opt/dremio/jars/3rdparty/:/var/dremio_efs/thirdparty/ com.dremio.dac.daemon.DremioDaemon

Executor
dremio 3287 1 12 Jan17 ? 11:55:46 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.amzn2.0.2.x86_64/jre/bin/java -Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler -Djava.library.path=/opt
dremio/lib -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/dremio/server.gc -Ddremio.log.path=/var/log/dremio -Ddremio.plugins.path=/opt/dremio/plugins -Xmx4096m -XX:MaxDirectMemorySize=24534m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/dremio -Dio.netty.maxDirectMemory=0 -Dio.netty.tryReflectionSetAccessible=true -DMAPR_IMPALA_RA_THROTTLE -DMAPR_MAX_RA_STREAMS=400 -XX:+UseG1GC -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC -cp /opt/dremio/conf:/opt/dremio/jars/:/opt/dremio/jars/ext/:/opt/dremio/jars/3rdparty/:/var/dremio_efs/thirdparty/ com.dremio.dac.daemon.DremioDaemon
ed2-user 26247 26171 0 13:22 pts/0 00:00:00 grep --color=auto dremio

24

@Diego Few things

  • After you have increased your direct memory to 24 GB, have you run the query in Job ID# 223529ef-69d5-5f6e-742d-cee7981b1c00 again?I yes, send me the profile, if not, please run it and see what happens, if it fails, send the profile again
  • You are on a very archived version of Dremio 3.x, please upgrade to ASAP as at some point, we are not able to support you just because the code has significantly changed and we cannot even troubleshoot this anymore
  • Memory requirements depends on # and type of queries you have, we can only advice on why a query ran OOM but if you are looking for answers like
  • “How many executor nodes do I need”?
  • What will be the direct/heap memory required for each node

These answers are not the same for every cluster and depends on the workload and you would have to work with an architect , again the version you are on does not have many things that are on 21.x, for example, Dremio 3.x does not have the concept of Engines where different workloads can be clubbed into it’s own pool of executors, reflection matching algorithms cover much more matching patterns, coordinator and Executor heap memory usage are far more optimized, if you are reading Parquet then the read performance of the Parquet files are much more optimized, I can keep going on but this is something, you need to take a step back and think about

Kindly let us know, if you have any other questions

25

@balaji.ramaswamy I’m on 22.1.1-202208230402290397-a7010f28 version of Dremio

26

@balaji.ramaswamy I created a new node and assigned it the preview queue. I removed the executor role from the master node. Apparently this is the solution to my problem.

image
image796×545 28.8 KB

27

Thanks for the update, yes, it is not recommended to run the master also as an executor. Good catch