Hey,
Does anyone have a good method for checking if any jobs (doesn’t matter what type) are running in Dremio? I don’t see a simple way of doing this via the API.
We use Azure scale sets for our executors and we want to prevent them from scaling down if there are any jobs running. We are considering using Prometheus to monitor CPU activity but I’m wondering if there is a better way.
Thanks
Zack
you can make a GET request to http://remio-host:9047/apiv2/jobs/ currently the UI of dremio is a simple react App so you can use chrome/firefox netwrk monitor to cacth request and use it as API
(some endpoints used by the UI are not docuemnted in API) be carefull
Awesome thanks dacopan.
I ran that api call in python and it returned the 100 oldest jobs stored in Dremio
So I modified the api call to:
jobs?filters=%7B"jst"%3A%5B"RUNNING"%2C"ENQUEUED"%5D%7D&order=DESCENDING&sort=st
decoded: jobs?filters={“jst”:[“RUNNING”,“ENQUEUED”]}&order=DESCENDING&sort=st
It now returns the 100 most recent jobs, but it seems to ignore the RUNNING, EN-QUEUED filter. Any idea why?
From my understanding, using the filters requires javascript to be executed. I don’t want to deal with that so I am just going to do –
jobs?limit=5000&order=DESCENDING&sort=st#
For my environment, 5000 jobs will cover the at least the past 48 hours of activity. So this will catch any jobs that are still RUNNING or ENQUEUED.
*For others: make sure you use apiv2 and not api/v3 for this jobs call…
Thanks,
Zack