We’ve got some jobs that use Arrow Flight to query and insert/delete data in iceberg tables.
We regularly get gRPC Resource Exhausted and ENHANCE_YOUR_CALM error responses.
We haven’t been able to identify the cause. Where can we look server-side to find it?
We’re using the .net Apache.Arrow.Flight library client-side, the exception from it is:
Grpc.Core.RpcException: Status(StatusCode=“ResourceExhausted”, Detail="Error starting gRPC call. HttpRequestException: The HTTP/2 server closed the connection. HTTP/2 error code ‘ENHANCE_YOUR_CALM’ (0xb). (HttpProtocolError) HttpProtocolException: The HTTP/2 server closed the connection. HTTP/2 error code ‘ENHANCE_YOUR_CALM’ (0xb).
We’ve noticed that we sometimes get the failing queries in the Dremio job log. They look like the client has cancelled after 3 minutes and 10-20 seconds.