Bigger picture: I think Dremio may be too powerful for the infrastructure we have, how to make sure I don’t crash either Dremio or flood the database server connections, threads, etc.?
Medium picture: I need to get a better understanding of how the Dremio JDBC connector queues reflections and other high cost queries. I see the Queue section of the admin portal provides some control over the number of nodes, but not the number of threads per query.
- In the Dremio settings can I control the parallelism of the queries themselves in a queue, as opposed to the number of concurrent jobs?
- Where does any record of the “queue” exist, that is the order in which staged queries will be sent along? Do I just have to sort jobs by when run, and reason it out?
I plan for production to limit the number of connections from Dremio using the database’s listener.ora settings.
- Is limiting Dremio to 8 or 12 threads out of 16x2 (32 total) too few?
- How should I time and size the queries and compute instances according to different loads? I have queries with a max size of 1.5GB running in around 30 minutes?
Details on the limitation I have observed
Under Dremio Jobs High Cost Reflection my Oracle Source database (Oracle 19c EE, 16 core Solaris) is throwing an error indicating there are too many parallel queries for the server to handle.
ORA 12801 error signaled in parallel query server P00L
Using development Oracle database I can use SQL Developer to monitor sessions and view 16 active aggregation reflection queries Dremio is sending in parallel, several more dremio sessions marked as inactive that appear to be recent query runs or attempts. I cannot kill sessions and need to prepare for logging of sessions and other OLTP overhead once moving to production. I am relying on Dremio to know not to overload the server, or let me control its queries somehow.
What to do??
- I must assume any queries that fail over the JDBC driver can leave interrupts or inactive pending connections.
- There appear to be several background refreshes to metadata and to check freshness of the reflections.
- I can control which AWS cluster (engine) each queue uses, I just don’t have control over the amount of parallelism in the queries themselves or an understanding of how the order works.
- As an outside-of-Dremio solution, would I just need to schedule reflections to run overnight to utilize more parallelism, use multiple database logins scoped to having different parallelism settings?