Timeout while waiting for Python Arrow Flight client

1

I’m using the Python arrow-flight-client as a means to access the dataset in Dremio

I am trying to access the data in an iteration, fetching 1000 records in each iteration, and randomly either at first or nth iteration I am getting this error in my local machine:

Exception: FlightInternalError('gRPC returned internal error, with message: RpcException: Timeout while waiting for client to be in ready state… Client context: OK. gRPC client debug context:

And getting this in Dremio UI
RpcException: Timeout while waiting for client to be in ready state.

I’m new to arrow flight / Dremio.I tried increasing timeout in flight.FlightCallOptions but it had no significant effect.

Can you please guide me if there is some other timeout to be set specifically for this gRPC error.

Thanks for your time and attention.

-Prashant

2

Hi Prashant,

You might be able to solve this by increasing the value of flight.client.readiness.timeout.millis. Which version of Dremio are you using?

3

Hi James,

Thanks for your response, currently I am using Build 19.0.0 and the library flight.client is depreciated in my case.

Along with the timeout issue I noticed a case where my Dremio is not taking more than one pyarrow query request at a time, i.e. when the same script is run in two parallel instances, the one which executes first gives no error but the second one waits for some time and gives the following error:
RpcException: Timeout while waiting for client to be in ready state.

Is there any way to allow pyarrow queries to have multiple instances the way it is for ODBC so multiple queries can be taken parallelly at a time.

Thanks.