Kubernetes elastic engine in Community Edition

1

Hi, following the docs and UI, I don’t see Dremio supports Elastic Engines using Kubernetes.

Do you have any plan to add this?

Thanks.

2

@chulucninh09 When you say elastic engines, is this for auto scaling or routing specific queries (via queues) to specific executors

3

@balaji.ramaswamy yes it is, on-prem version in general doesn’t provide support for Kubernetes as the following screenshot. Is only YARN is supported right now?

image
image781×733 29.8 KB

4

@chulucninh09 For K8’s you have to go via the docker image and Helm charts

5

@balaji.ramaswamy as my understanding, k8s is the deployment model for dremio cluster.

My concern is how can I create engines with k8s and auto start/stop those engines for particular queues and workload?

6

@chulucninh09 Auto stop and start is only available in the AWS edition but you can still direct queries to different engines in K8’s, let me get back on that

7

Hi Balaji,

Is it possible to have elastics engines when Dremio is deployed on Kuberenetes using AKS or EKS

8

@unni Do you need to route queries to specific standalone executors?

9

@unni : I can see we can create multiple engines by changing this variable in helm chart values file. Which I think is the elastic engine feature mentioned here. I tried this and it works fine. I am using dremio community but since we don’t have workload management feature in dremio community. I couldn’t leverage these engines.
@balaji.ramaswamy : Is there a way we can make use of these engine using queue control which is the only similar option in dremio community ?

10

@irshad-pai

Good point, engines/WLM is an Enterprise edition feature only

@unni Are you on CE or EE?