We are currently running Dremio 3.0.6 in distributed mode through plain docker (i.e no kubernetes) and trying to upgrade to dremio 3.1 also using plain docker.
The steps which we’ve used to deploy 3.0.6 were the ones in the docker hub at the time (screenshot attached).
We are using the same steps to deploy Dremio 3.1 and looks like Dremio only starts as master coordinator and executor. We are unable to start coordinator and executor separately despite setting the java environment properties same as we did for 3.0.6.
Is there anything else that needs to be set for deploying dremio 3.1 using docker only?
For each node type have, you tried explicitly disabling/enabling each type? For instance, on the master coordinator:
Also, what upgrade steps are you following here?
We got it to work now.
Instead of passing the parameters via command line options to docker run as we did in 3.0.6, we set the DREMIO_JAVA_EXTRA_OPTS environment variable to contain the parameter values and everything starts properly now.
Below is what worked:
docker run -p 9047:9047 -p 31010:31010 -p 45678:45678 -e DREMIO_JAVA_EXTRA_OPTS="-Dzookeeper=my_zk_quorum -Dservices.coordinator.master.embedded-zookeeper.enabled=false -Dservices.executor.enabled=false" dremio/dremio-oss
docker run -p 9047:9047 -p 31010:31010 -p 45678:45678 -e DREMIO_JAVA_EXTRA_OPTS="-Dzookeeper=my_zk_quorum -Dservices.coordinator.master.enabled=false -Dservices.executor.enabled=false" dremio/dremio-oss
docker run -p 45678:45678 -e DREMIO_JAVA_EXTRA_OPTS="-Dzookeeper=my_zk_quorum -Dservices.coordinator.enabled=false" dremio/dremio-oss
Thanks for sharing your solution @igreg. We are definitely encouraging people to use K8s for cluster deployments, and we won’t be documenting multi-docker configurations going forward. If you still want to use the latter, you should look to our template Helm charts for guidance on how to “manually” configure the cluster with Docker.