When submitting a query to Dremio using Flight, is there anyway to get the jobID that’s generated for the query? I’d like to use this information to retrieve status info while the query is running. I’m looking for the ID that can be passed to the REST API to retrieve job information.
I don’t think you can get the job id from the Flight SQL client. You might be able to manually get the jobid since you know the user, SQL and approximate time. The UI issues search queries against the indexed job collection. You could use browser dev tools to see the REST API call and probably craft your own REST query to get the job details.
Yes, this is the method we use currently, but it bangs up the master node pretty hard with a lot of load. Starting in version 15.0 of Flight, they added an app_metadata field to the FlightInfo message (see [FlightRPC] FlightSQL: Expose custom query metadata · Issue #37635 · apache/arrow · GitHub).
If I put up a PR to add the query ID to the app_metadata field, is there any chance it would be accepted?