we have Dremio production Env and setup one master and 4 coordinator nodes but daily bases we are getting Jolokia memory heap usages alerts so how we can fix.
Please suggest quick action really appreciated
Hi Ben,
Thanks for your response
.Now we have set the threshold limit 85 percentages and the issue came within 24 hours 15 -20 time. How we can fix and why this is coming frequently.
Also one more query if we are getting out of memory issue comes then how we can resolve and what things we need to check in our side. Please suggest
Thanks in advance.
Regards,
Manish.
@Manish,
Dremio will periodically reach the maximum heap limit and then the JVM will reclaim this memory in the usual garbage collection cycle. This is not necessarily a problem. If you are receiving Out Of Memory errors, then you either need to change your queries or scale you resources.
How much memory, heap and direct, have you allocated to your executors and coordinator?
Here’s more information about memory configuration from our documentation: https://docs.dremio.com/deployment/memory-config.html
@ben: sorry for my side as of now we are using one master and four executors and getting frequently memory heap issue and out of memory issue.
As you suggest the Heap memory is not an issue but for out of memory we need to check our query or update the resources also I will send my memory configuration asap then please suggest what we need to do on this situation. Thank you for your quick responses.
Hi Ben,
max.direct.memory.executor 102400
max.direct.memory.master 45056
max.heap.memory.executor 16384
max.heapmemory.master 16384
Is it any changes required ?
Hi @ben
One query I have, could you please let me know why most of the time only executor 3 nodes took maximum heap memory. As of now, we have 4 Executors.
Could you please let me know your ideas about this.
I would actually reduce the heap on the executor to only 4 GB. But the problem likely is in the query itself. Some operation is too heap expensive.
Can you share the profile for the job?