As one of the core capabilities of Dremio is to provide end users a self-service capability by way of creating Virtual Data Sets, a concern arises that these users who may not be SQL savvy may inadvertently create VDSs that are built on high cost SQL queries - even unintentional Cartesian products could result. What is Dremio’s solution, if any, to this possibility in the realm of monitoring and preempting such occurrences? Is Dremio dependent on the various underlying data sources timeout mechanisms or external monitoring tools to preempt such occurances?
Hey Lew, thanks for the question! We’re planning to address such scenarios as a part of our workload management work in our Enterprise Edition (more updates coming soon). As we release more capabilities here, admins will be able to manage this within Dremio. Over time, the idea is to provide a rule-based system where users can target queries based on operations in the query (full scan, cartesian join, no predicates, etc.) as well as run-time metrics (run-time, rows scanned, rows sorted, memory spilled/user, etc.). They can use this information to automatically reject or place queries into queues, as well as cancel active queries.
1 Like
Hello,
I have the same question, now we are on version 20 Enterprise Edidition. How can we reject the “bad query” based on operations in the query or run-time metrics as you said @can ?
There are not documentation about this.
thanks
@hieuiph Via WLM you can set max job and queue memory limits, You can also set Max runtime limit. This way you can avoid log running full table scan queries that may use up all the memory available
Hello @balaji.ramaswamy , thanks for your answer. I tried all you suggested but it doesn’t work if a user run the “bad query” like “Select distinct * …”. I set up limit query max =1, queue memory limits = 23 (max memory on every executor = 37GB, we have 4 executor).