Timeout while waiting for (Golang Arrow Flight) client

1

I’m using the golang arrow-flight-client example.go file on dremio-hub as a basis for programmatic access to virtual datasets I have created in dremio.

The physical dataset is ~800 million records. My virtual dataset is ~46 million. I can successfully run this query from my client when dremio is running locally, but when I access a dremio running on a remote server, I get the following error:

(com.dremio.exec.rpc.RpcException) Timeout while waiting for client to be in ready state.

I’m new to arrow flight / dremio. Is there some grpc timeout that should be set when creating my flight client?

Thank you for your time and attention to this matter.

-Benjamin

2

@grossb1

Can you please try to set the support key flight.client.readiness.timeout.millis to a higher value?

The problem here is that the client is trying to load all records into memory before printing them out. For large numbers of records, this can get quite slow in the JVM resulting in the client timing out.

3

Thanks for the guidance, increasing this setting allows the query to run to completion.

Best,
Benjamin

4

@grossb1 Glad it worked ! Thanks for the update

5

Can you help me , how to set this property flight.client.readiness.timeout.millis. I am using Java Arrow flight-core 3.0.0 . I am getting following error …
‘[ERROR] Exception: RpcException: Timeout while waiting for client to be in ready state.
org.apache.arrow.flight.FlightRuntimeException: INTERNAL: RpcException: Timeout while waiting for client to be in ready state.
at org.apache.arrow.flight.CallStatus.toRuntimeException(CallStatus.java:131)
at org.apache.arrow.flight.grpc.StatusUtils.fromGrpcRuntimeException(StatusUtils.java:152)’
Thank you .

6

Hi,

Do the following:

Admin > Support and type the parameter in the text box to the bottom left and click show. The default is 5000ms

7

Hi @balaji.ramaswamy
Tow question for this configuration.

What will affect of this timeout setting value? This setting will affect max read time of one batch data or the max read time of all data in one query?

How can we decided the exactly value of this setting? Based on result set record size and the number of query fields?

8

@Tony.Zhang Only if you are reading large number of records then yes, values should be increase only if you timeout and depends on how many records are returned

9

Can this parameter be updated programmatically? Using the REST API or passing it as a parameter in the flight request

10

@gpiguing

You should be able to set at the system or session level via a SQL query. Something like:

ALTER SYSTEM SET flight.client.readiness.timeout.millis = 5000;

OR

ALTER SESSION SET flight.client.readiness.timeout.millis = 5000;

Use the SQL API.

11

is there a different setting that would cause the same query that works from the UI to fail when run via Arrow Flight client? the jobs always get cancelled at 60 seconds when run via Arrow Flight but work just fine in the UI. setting: flight.client.readiness.timeout.millis to the max 900000 didn’t make a difference.
any hints?

12

In Work load management, there is query runtime limit, depends on which queue your arrow flight request will be assigned to

13

Thanks for the reply, i am using the open source edition and see the value Maximum query memory set to 0, which means unlimited. the query size doesn’t seem to matter when run from Arrow, it always times out after 60 seconds, the actual runtime from the UI, which works for the same query, is just over 2 minutes.

14

Just to round this convo to completion - the UI will truncate result sets to 1M records. This may well be the cause here see Dremio

16

I deleted my post by accident.

Just a note.

In case anyone is using helm chart to deploy Dremio, there’s the option to set this property via extraStartParams.

extraStartParams: >-
  -Ddremio.debug.sysopt.flight.client.readiness.timeout.millis=900000

It is working for me :slight_smile:

image
image1020×477 28.2 KB