DREMIO_MAX_HEAP_MEMORY_SIZE_MB and DREMIO_MAX_DIRECT_MEMORY_SIZE_MB setting is lost when the executor nodes are stopped. Do we have an option to maintain this config when existing executor nodes are stopped and new executor nodes are created?
Note: Currently we are using elastic engine setup for executor nodes and stops every 5 min when idle.
I assume you are on the AWS edition. Have you pushed the customization via the below method?
Customizing the executor, if /var/dremio_efs/executor_customization.sh exists on the EFS volume and is an executable, then the script will be run after the EFS volume is mounted and before Dremio is started. When restarting the executors, you need to restart the cluster not just bounce the Dremio process.
On the Coordinator after a project is created.
sudo mkdir -p /var/dremio_efs/executor/etc/dremio/
echo “#!/bin/bash” > /var/dremio_efs/executor_customization.sh
sudo chmod 755 /var/dremio_efs/executor_customization.sh
(re)start the executors
Changes to dremio.conf, dremio-env, logback.xml, core-site.xml etc
All changes that say to edit the dremio.conf file need this.
- On the coordinator, add the files you would like to override in /etc/dremio/ on executors to /var/dremio_efs/executor/etc/dremio/
- On the coordinator, add the following line to /var/dremio_efs/executor_customization.sh and restart the executors
cp -rf /var/dremio_efs/executor/etc/dremio/* /etc/dremio