I’ve developed a web app that displays a lot of data. To speed up the initial loading time of this data, I’ve integrated concurrency into the application. In other words, instead of sending 1 query after another, I send all the queries at the same time.
Are there Dremio limitations on concurrency? I noticed this popup when I deployed the app.
Does this mean it’s better to run queries synchronously versus concurrently?
This and other observations are determined per job based on the job’s stats. This means that a relatively high percentage of the run time was spent waiting for the CPU, suggesting the executors were oversubscribed. There are no hard limits on concurrency or hard rules on what level of concurrency is optimal. All of that will depend on the workload and available resources. I definitely would not say it’s better to run queries synchronously. However, there is a tipping point where too much concurrent work can cause things to slow down as different queries vie for the same limited resources. One or two queries having proportionally high CPU sleep are non-problematic. However, continuous stretches where many queries show this suggest it is time for more CPU cores, at least during those periods.