Hash exceeded the memory limits

1

I’ve receiving the next message for simple hash aggregation query that I expect to be fit to memory. No other queries are running in the background.
I’m using 1TB RAM and Dremio is set to use >800 GB Direct Memory (see java flags bellow).
I’m building a table expected to be 200 GB.

Is it possible that Dremio is using less memory that he can?

Exception:
ChannelClosedException: [FABRIC]: Channel closed null <–> null (fabric client)

server.log:
2021-08-03 07:05:45,323 [e27 - 1ef717a7-58b6-9bfa-4a39-2b531860b200:frag:3:40] ERROR com.dremio.sabot.driver.SmartOp - Query was cancelled because it exceeded the memory limits set by the administrator.
com.dremio.common.exceptions.UserException: Query was cancelled because it exceeded the memory limits set by the administrator.

Query profile zip:
aggregation_orders.zip (114.9 KB)

Java flags:
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler
Djava.library.path=/opt/dremio/lib
XX:+PrintGCDetails
XX:+PrintGCDateStamps
Xloggc:/opt/dremio/log/server.gc
Ddremio.log.path=/opt/dremio/log
Ddremio.plugins.path=/opt/dremio/plugins
Xmx8192m
XX:MaxDirectMemorySize=862208m
XX:+PrintClassHistogramAfterFullGC
XX:+HeapDumpOnOutOfMemoryError
XX:HeapDumpPath=/opt/dremio/log
Dio.netty.maxDirectMemory=0
Dio.netty.tryReflectionSetAccessible=true
DMAPR_IMPALA_RA_THROTTLE
DMAPR_MAX_RA_STREAMS=400
XX:+UseG1GC
cp /opt/dremio/conf:/opt/dremio/jars/:/opt/dremio/jars/ext/:/opt/dremio/jars/3rdparty/* com.dremio.dac.daemon.DremioDaemon

2

@shai.bet Probably nblab36.neuroblade.corp is doing a Full GC and went unresponsive, are you able to send us all the GC logs from the log folder?

3

Attached GC file with new profile file.
Will the GC cause out-of-memory exception? I can understand why it causes the closed channel.

BTW, Is there a difference of memory handling (spill and such) between HashJoin and HashAgg?
I know that HashJoin do not spill to disk.

aggr_orders_logs.zip (2.7 MB)
aggr_orders.zip (109.1 KB)

4

@shai.bet Is the GC file from the coordinator? We need it from the executor, chck IP address in error message of job ID 1ef5c3a3-8a5f-4371-0c39-a211ae53f200 along with the server.log when this query ran which is 2021-08-04 07:06:03 so it will be under the log/archive folder and name should be server.2021-08-04.0.log.gz, if there multiple files on the 4th, send theem all