Hash exceeded the memory limits

I’ve receiving the next message for simple hash aggregation query that I expect to be fit to memory. No other queries are running in the background.
I’m using 1TB RAM and Dremio is set to use >800 GB Direct Memory (see java flags bellow).
I’m building a table expected to be 200 GB.

Is it possible that Dremio is using less memory that he can?

Exception:
ChannelClosedException: [FABRIC]: Channel closed null <–> null (fabric client)

server.log:
2021-08-03 07:05:45,323 [e27 - 1ef717a7-58b6-9bfa-4a39-2b531860b200:frag:3:40] ERROR com.dremio.sabot.driver.SmartOp - Query was cancelled because it exceeded the memory limits set by the administrator.
com.dremio.common.exceptions.UserException: Query was cancelled because it exceeded the memory limits set by the administrator.

Query profile zip:
aggregation_orders.zip (114.9 KB)

Java flags:
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler
Djava.library.path=/opt/dremio/lib
XX:+PrintGCDetails
XX:+PrintGCDateStamps
Xloggc:/opt/dremio/log/server.gc
Ddremio.log.path=/opt/dremio/log
Ddremio.plugins.path=/opt/dremio/plugins
Xmx8192m
XX:MaxDirectMemorySize=862208m
XX:+PrintClassHistogramAfterFullGC
XX:+HeapDumpOnOutOfMemoryError
XX:HeapDumpPath=/opt/dremio/log
Dio.netty.maxDirectMemory=0
Dio.netty.tryReflectionSetAccessible=true
DMAPR_IMPALA_RA_THROTTLE
DMAPR_MAX_RA_STREAMS=400
XX:+UseG1GC
cp /opt/dremio/conf:/opt/dremio/jars/:/opt/dremio/jars/ext/:/opt/dremio/jars/3rdparty/* com.dremio.dac.daemon.DremioDaemon

@shai.bet Probably nblab36.neuroblade.corp is doing a Full GC and went unresponsive, are you able to send us all the GC logs from the log folder?

Attached GC file with new profile file.
Will the GC cause out-of-memory exception? I can understand why it causes the closed channel.

BTW, Is there a difference of memory handling (spill and such) between HashJoin and HashAgg?
I know that HashJoin do not spill to disk.

aggr_orders_logs.zip (2.7 MB)
aggr_orders.zip (109.1 KB)

@shai.bet Is the GC file from the coordinator? We need it from the executor, chck IP address in error message of job ID 1ef5c3a3-8a5f-4371-0c39-a211ae53f200 along with the server.log when this query ran which is 2021-08-04 07:06:03 so it will be under the log/archive folder and name should be server.2021-08-04.0.log.gz, if there multiple files on the 4th, send theem all