Hi,
I just need a clear cut understanding on the memory allocation to discuss with the team. I am currently working with the latest helm chart of dremio. In the chart, it is given that, for executor
if memory request <= 32786, then heap is 8192
if memory request <= 6144, then heap is 4096
else heap is 2048
Only 3 values are possible for heap. And we can override this with environment variable, DREMIO_MAX_MEMORY_SIZE_MB. But when i give that, both executor and master heap is changed to the value provided in the environment variable. I have tried will all these combinations, and getting heap errors.
My question is,
1. why did the criteria came for these 3 particular values. Is there any significance for these.?
2. For 1 and 3 rd option mentioned above, the helm chart is taking an exact difference from tottal request to the heap as direct memory. That is, as per chart, for 1 and 3 option, “direct memory+heap=total memory request”. This violates the above point, that is leave some for os. We can only assume that the pod will be getting the requested resource only. But for 2nd option, actually this is not the case and there is a 2GB of memory left for usage other than direct and heap.
3. Why is the variable setting effecting both master and executor together. As per the documents, it is not the case with dremio.
4. How exactly should i handle the heap, direct and other memory for dremio master and executor pods for better performance.?
5. Is there any ratio between all these memory which affects the dremio usage and performance.?
I have modified the helm chart to change the heap and direct memory as per my requirement via chart values file. Even now i have heap related issues.
6. Why is there such an option to handle this dynamic heap memory change.?
7. How can i debug these issues, since i am not able to find any logs regarding these in dremio pods.
can you provide a brief idea of all these topics, so that i can fine tune my spec for dremio for proper working.