Dremio on Docker Hub - exceeded the memory limits

1

I instantiated an ec2 t3a.2xlager (8cpus and 32 GB) with ubuntu server 18. I installed the docker and from it the official image of the dremio that is in the docker hub.

I was working normally and everything was working perfectly, but I created a new query and to execute it, Dremio returns me the message "Query was canceled because it exceeded the memory limits set by the administrator. Expected at least 10.03GB bytes, but only had 8.00GB available. "

How to increase this memory limit? Remembering that I have the dremio from the docker and I have not found how to change it with the dremio.conf settings

I’ve used the command sudo docker run -it --memory = 28G -p 9047: 9047 -p 31010: 31010 -p 45678: 45678 dremio / dremio-oss --cpu 8

And when I run docker stats I confirm that the 28GB is available for this dremio container, however some internal configuration limits it to 8GB.

Thank you

2

@DavidAndrade

You have to change it in the values.yaml file, see documentation for EKS and AKS, should be similar in any K8’s deployment

http://docs.dremio.com/deployment/amazon-eks/eks-custom.html#cluster-configuration
http://docs.dremio.com/deployment/azure-aks/aks-custom.html#cluster-configuration

3

Guys, I managed to solve the problem. Here is the solution:

  1. First look for the configuration file on your linux with the command: sudo find. -iname dremio.conf

  2. Enter the directory he is in

  3. open the dremio-env file for editing

  4. at the memory location, change it according to your need, mine was like this:

Max direct memory size (in MB) for the Dremio process

Default to 8192 for server

DREMIO_MAX_DIRECT_MEMORY_SIZE_MB = 28192

  1. save the file and restart the dremio container.

Thank you all !

1 Like