Helm Direct Memory Bug

gpdenny · June 7, 2021, 11:57am

When setting the executor memory to 6144 using the provided Helm chart DREMIO_MAX_DIRECT_MEMORY_SIZE_MB is set to 0 and Dremio fails to start.

I believe there is a bug in the chart when calculating the direct memory size in relation to the max heap memory size.

Not sure how welcome contributions are, however, I have opened a PR here for review which I believe resolves this bug:

github.com/dremio/dremio-cloud-tools

fix: V2 chart executor direct memory size

dremio:master ← gpdenny:executor-direct-mem

opened 05:29PM - 28 May 21 UTC

gpdenny

+1 -1

**Description:** Bugfix, non-breaking change which fixes an issue. Default `DR…EMIO_MAX_DIRECT_MEMORY_SIZE_MB` not applied correctly. **Test** ``` # Dremio Executor executor: # CPU & Memory # Memory allocated to each executor, expressed in MB. # CPU allocated to each executor, expressed in CPU cores. cpu: 2 memory: 6144 ``` **Expected Behaviour:** ![Selection_990](https://user-images.githubusercontent.com/37297485/119980621-71407400-bfb4-11eb-9c0a-732cc17a2f7d.png) _(https://docs.dremio.com/deployment/memory-config.html)_ `DREMIO_MAX_HEAP_MEMORY_SIZE_MB` is 4096 and `DREMIO_MAX_DIRECT_MEMORY_SIZE_MB` is 2048. **Actual Behaviour:** `DREMIO_MAX_DIRECT_MEMORY_SIZE_MB` is 0 for executors. This causes the Dremio executor to fail on startup. ``` env: - name: DREMIO_MAX_HEAP_MEMORY_SIZE_MB value: "4096" - name: DREMIO_MAX_DIRECT_MEMORY_SIZE_MB value: "0" ``` ``` dremio-executor 2021-05-28 11:43:15,838 [main] ERROR ROOT - Dremio is exiting. Failure while starting services. dremio-executor java.lang.NoClassDefFoundError: Could not initialize class io.netty.buffer.PooledByteBufAllocator dremio-executor at org.apache.arrow.memory.RootAllocatorFactory.checkPoolConfiguration(RootAllocatorFactory.java:83) dremio-executor at org.apache.arrow.memory.RootAllocatorFactory.newRoot(RootAllocatorFactory.java:78) dremio-executor at org.apache.arrow.memory.RootAllocatorFactory.newRoot(RootAllocatorFactory.java:56) dremio-executor at com.dremio.exec.server.BootStrapContext.<init>(BootStrapContext.java:66) dremio-executor at com.dremio.dac.daemon.DACDaemonModule.bootstrap(DACDaemonModule.java:261) dremio-executor at com.dremio.dac.daemon.DACDaemon.<init>(DACDaemon.java:185) dremio-executor at com.dremio.dac.daemon.DACDaemon.newDremioDaemon(DACDaemon.java:313) dremio-executor at com.dremio.dac.daemon.DACDaemon.newDremioDaemon(DACDaemon.java:321) dremio-executor at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:145) ``` I think the intended behavior was to subtract the heap memory size (4096 based on `6144 le engineMemory`), so direct uses the remaining?

Thanks!

balaji.ramaswamy · June 8, 2021, 5:41am

Thanks @gdelaney Will pass it on to the relevant team

Topic		Replies	Views
Dremio memory management issues	2	218	May 14, 2024
Configuring Dremio Memory Limits on Kubernetes	3	865	November 30, 2022
How to configure dremio-env file with a k8s installation of dremio?	2	741	February 6, 2023
How does memory allocation work while running reflections on dremio	1	464	December 1, 2023
Dremio using low memory	1	991	January 23, 2019

Helm Direct Memory Bug

Related topics