How does memory allocation work while running reflections on dremio

1

I am trying to create a reflection on a table that has almost 52GB in size. My dremio cluster has 8 executors each having 6 cores and 30gb memory. I am getting the error

Query was cancelled because it exceeded the memory limits set by the administrator.

I tried increasing the executor count to 12 and still getting the same issue. I even tried increasing heap memory by providing the following in my dremio-env file

DREMIO_MAX_HEAP_MEMORY_SIZE_MB=20480

Even giving 20GB as heap memory gives me the same error. How can i make sure my reflection runs? How does dremio use heap memory and direct memory for query execution and reflection runs?

2

Hi @sanchitsk,

You probably run out of direct memory, not heap. What’s the size of the direct memory on the executors?

Can you attach the reflection profile to the ticket, along with dremio.conf and the output of ps -ef | grep dremio?

Thanks, Bogdan