Dremio 11 - Aggregation Reflections memory limit

1

Hello, guys! I’m testing Dremio 11.00 AWS edition and I’m constantly getting this error message when trying to create Aggregation Reflections on my dataset. The strange thing is that on 4.9.3 or earlier I wasn’t getting this error and could create this reflection without any issues.

I’m joining a fact table (fato_vendas) with its dimensions (depara_lojas, depara_produtos, depara_tempo) in order to create a flat view (fato_vendas_enriquecida) to make me able to create its aggregation reflection with all needed dimensions.

image
image1842×1011 87.2 KB

My reflection list:

image
image1611×423 44.9 KB

The elastic engine responsible for creating reflections is made by 2 x m5d.8xlarge (256GB memory total).

Here’s the High Cost Reflections queue settings:

image
image450×940 27 KB

Here’s the query profile.
8d31dd35-efa8-4c8f-92d6-cb373848f89f.zip (55.7 KB)

One thing I noticed is that the node that is triggering this memory error is the master coordinator node but I already tried changing this machine type to the largest one available: m5d.8xlarge, with no success.

Is there anything I can do to fix this?

2

@mendeel

You are about to run out of heap memory and instead of the executor crashing we killed the query. We need to find out if you are sized right

You were already using 80 GB of direct memory on each of the executor when the query was killed, are you able to add the below parameters to dremio-env and send us the GC logs after retrying and if it fails with the same error?

DREMIO_JAVA_SERVER_EXTRA_OPTS=“-XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC -XX:+UseG1GC -XX:G1HeapRegionSize=32M -XX:MaxGCPauseMillis=500 -XX:InitiatingHeapOccupancyPercent=25”