We regularly upload parquet files to Azure Blob storage and use COPY INTO to copy the data into an Iceberg table.
We’re using Arrow Flight for the connection from which we execute the COPY INTO statement.
The Parquet files can be in the 100s of MB containing millions of records.
We regularly get a gRPC exception on the client side saying 420 ENHANCE_YOUR_CALM ( ENHANCE_YOUR_CALM error response - Apache Iceberg - Dremio)
In the job log it looks like the client cancelled the operation. It seems to be always around 3m15s (+/- 10 seconds or so) that the job is cancelled.
There is nothing in our client code that cancels, and as written above, the client gets an exception from gRPC.net.
Running the same COPY INTO query from the Dremio UI is successful.
We’re running 25.2.18 in Kubernetes.
Could there be some timeout that is triggered at around 200-220 seconds in the Flight connection handler if there is no data transferred between the Dremio Coordinator and the client?
Can it be configured, or is there some way to keep the connection alive?