I notice that there are queries stuck in dremio from time to time.
These queries are part of my batch ingestion pipelines.
Now that some are stuck, some of my pipelines never complete in full.
Can you advice what’s the possible reason?
Attached is the profile
fbf9ad4d-0290-4435-8d3e-74668247d1bd.zip (25.2 KB)
@Ken Seems like all phases are FINISHED, I assume the server side is complete
What client is triggering this? Example Airflow
If you still have the log file from the master log, are you able to send it? for “2025-01-08”
Yes, the client is airflow.
I am using Airflow for data ingestion into Dremio.
=========
I no longer have said log file, but I am experiencing the issue on a daily basis.
04c7cf59-6845-46c8-9994-ff179a76c0a4.zip (25.3 KB)
bc4348fe-3b07-47ba-a68b-3465d4b8174c.zip (25.2 KB)
Just thinking off the top of my head. Could this be due to insufficient memory for dremio?
From Airflow’s side, I can only see that the job got stuck.
Looks like usually it’s the COPY CSV FILE which got stuck.
—> Is it possible to set a timeout in the Dremio Client?
I am using this code snippet here for the python connector to Dremio
