Hi team,
I have below scenarios:
1)start dremio with 1 executor
2)As my load increase I need dremio to automatically scale out.
3)As load decrease dremio to reduce to 1 executor.
4)if I configure Dremio with max and min number of executors ,while starting dremio will it start with all max executors?
5)Is this possible in any mode of dremio deployment? Is this possible with AKS deployment? Do we have any document in detail about auto scaling? I couldn’t get a complete picture from the elastic engine document.
@ann Even if you configure this on the AKS side, when a scale up happens, queries that have already entered planning and ready for execution or have started execution will not leverage the new nodes, if you scale down, existing queries may fail as fragments might be active on an executor pod that was brought down
@balaji.ramaswamy does that mean autoscaling is not currently supported in dremio?
I am bit confused by reading documentation about auto scaling
Here it is mentioned to specify instance count(eg:5). So when starting dremio after installation, with how many nodes it will start?
When we issue a query will dremio start with all 5 Nodes?
How exactly autoscaling work here?
@ann I do not find the word “auto scale” in the document. I also do not see “mentioned to specify instance count”, can you please copy/paste the exact line you are talking about?