Hi, following the docs and UI, I don’t see Dremio supports Elastic Engines using Kubernetes.
Do you have any plan to add this?
Thanks.
@chulucninh09 When you say elastic engines, is this for auto scaling or routing specific queries (via queues) to specific executors
@balaji.ramaswamy yes it is, on-prem version in general doesn’t provide support for Kubernetes as the following screenshot. Is only YARN is supported right now?
@chulucninh09 For K8’s you have to go via the docker image and Helm charts
@balaji.ramaswamy as my understanding, k8s is the deployment model for dremio cluster.
My concern is how can I create engines with k8s and auto start/stop those engines for particular queues and workload?
@chulucninh09 Auto stop and start is only available in the AWS edition but you can still direct queries to different engines in K8’s, let me get back on that
Hi Balaji,
Is it possible to have elastics engines when Dremio is deployed on Kuberenetes using AKS or EKS
@unni Do you need to route queries to specific standalone executors?
@unni : I can see we can create multiple engines by changing this variable in helm chart values file. Which I think is the elastic engine feature mentioned here. I tried this and it works fine. I am using dremio community but since we don’t have workload management feature in dremio community. I couldn’t leverage these engines.
@balaji.ramaswamy : Is there a way we can make use of these engine using queue control which is the only similar option in dremio community ?
Good point, engines/WLM is an Enterprise edition feature only
@unni Are you on CE or EE?