Dremio, Docker, and Kubernetes

1

Note: Since this was posted, we now have published helm charts to deploy Dremio to Kubernetes at https://github.com/dremio/containers/tree/master/charts/dremio . It is recommended to use them to deploy to Kubernetes.

We recently published some experimental tools to help you deploy Dremio clusters. This includes both tools to help you build Docker images, as well as Kubernetes templates to deploy Dremio clusters in any configuration you like.

You can find these on github: https://github.com/dremio/containers

Note these tools work for both Community and Enterprise Editions.

As an example, here’s how you would provision a cluster with one master coordinator and 5 executor nodes.

kubectl create -f zookeeper.yaml
kubectl create -f dremio-configmap-minimum.yaml
kubectl create -f dremio-master-volume-hostpath.yaml
kubectl create -f dremio-master-volume-pvc.yaml
kubectl create -f dremio-master.yaml
kubectl create -f dremio-service-ui.yaml
kubectl create -f dremio-service-client.yaml
kubectl create -f dremio-executor.yaml
kubectl scale --replicas=5 rs/dremio-executor

Many of you may have enhancements or additions to these tools. We welcome your pull requests!

1 Like
3

Hi, yes they are up to date. What issue are you running into? One issue that is typically seen is that pods await scheduling due to lack of resources. The default CPU and memory in these are set to what the Dremio documentation states. Can you check the pod statuses and see what is holding them up? If it is resources, you can try reducing the cpu and memory settings in values.yaml.

4

The other thing to look at would be the helm charts we publish to deploy in kubernetes: https://github.com/dremio/containers/tree/master/charts/dremio

6

Do you mind sharing your findings as it may help others with the similar issue? Thanks.

7

Hello, I followed the instructions and tried to deploy the Dremio by cloning it’s github repo. But unable to deploy it , as it is showing error: error parsing dremio-executor.yaml: json: line 0: invalid character ‘{’ looking for beginning of object key string.

8

Hello @garimatanupatel,

Can you share some more info? What was the helm command you ran? Your values.yaml file.

9

I am not using helm for same. I m using these commands only .kubectl create -f zookeeper.yaml
kubectl create -f dremio-configmap-minimum.yaml
kubectl create -f dremio-master-volume-hostpath.yaml
kubectl create -f dremio-master-volume-pvc.yaml
kubectl create -f dremio-master.yaml
kubectl create -f dremio-service-ui.yaml
kubectl create -f dremio-service-client.yaml
kubectl create -f dremio-executor.yaml
kubectl scale --replicas=5 rs/dremio-executor.
Is it mandatory to use helm for the same.

10

@garimatanupatel,

You can do it without helm but then you need to ensure that the yaml files generated or written are correct. And you run them in the correct order. From your error, you have some error in your dremio-executor.yaml file.

As is mentioned in the thread, we do publish helm charts at https://github.com/dremio/containers/tree/master/charts/dremio that definitely makes life lot easier.