I 'm trying dremio 24.1 community edition for my project.
I have some slow query and concurrent use cases.
At the beginning, I use 1 master node and 1 executor node for test to get the performance. Then I try to add 1 more executor node to my cluster but no help on the performance. Could I know that what’s wrong ?
Hi @fiona, there could be a number of reasons for this behavior. Is it possible for you to share the job profile so we can get a closer look at what might be happening?
There are some security policy here so that I may not able to share this to you directly. Cloud you kindly share me some common reasons for that? I could provide some more details for your reference if needed.
@fiona Scale out coordinator would help only if time was spent in command pool due to too many queries submitted at the same time
- How many cores does your coordinator have?
- Where is the time spent, the job profile (raw profile) should tell you that
Hi Balaji, Thanks for your response.
For concurrent query:
My servers are 16 core 256 G and notice that when trigger more than 20 concurrent queries to cluster the performance will go down.
Scale out coordinator would help only if time was spent in command pool due to too many queries submitted at the same time.
Do you mean that I should try 2 coordinator and 1 executor for my concurrent test to see any improvement or not? My understanding is not like that, below is the doc from dremio:
For single query:
Do you mean that if the cluster resource is enough, scale up executor node is not the way of accelerate query? I know that we could have reflection or partition or any other way to accelerate query . Just want to understand more about in what condition we could scale up the executor to get better performance.
@fiona Any chance you can send us the raw profile? We can exactly pin point where the bottleneck is
On the jobs details page on the bottom left, click on “Download Profile” and send over the downloaded zip file