Getting this error: RpcException: Timeout while waiting for client to be in ready state

Hi everyone,

we have multiple dashboard on qlik they are scheduled at 1 hour gap, So dremio is running the query every hour but sometimes it gives this error after 4 or 5 mins:
RpcException: Timeout while waiting for client to be in ready state.
And qlik goes for 3 hours because of this error.
I have also tried solution suggested by @balaji.ramaswamy where I have configured
this property from 50,000 to 120000.
image

and it is still not working
So @balaji.ramaswamy let me know what should I do to solve this, we are using dremio comunity and facing this error in production.

This is the profile for that query
91841e63-f93d-48d4-842d-ee6d3cc8c1fd.zip (24.8 KB)

Please let me know what is the solution for this as soon as possible.

Thank you,
Fenil Jain

Hi @balaji.ramaswamy, If you can check the query profile then please let me know the solution.

@fenilj

You should investigate what is happening on the client side. The timeout is 50 seconds to process each batch returned by the server which is quite long and you have increased to 120s. It looks like you are sending 4,916,352 records and close to 1 GB of data. Can you try and see if sending a subset of this dataset also times out?

@balaji.ramaswamy This is the profile of different data set where records are less and getting same error
42694f8b-ebbc-4540-b78b-26a1a6c5cfb7.zip (48.6 KB)

@fenilj Have you tried with a different client. Wondering if this something specific to this client?

@balaji.ramaswamy You mean BI tool? Currently we are using qliksense, and just for your information, it was working fine before, we are getting this error since last 3 weeks.

And I have also provided this solution in qlik, with each query we are connecting and disconnecting.

@fenilj That is a great data point that this issue has been happening fot the last 3 weeks. Did you change anything on the Dremio side (server) like version upgrade? driver change? Was flight introduced 3 weeks back?

Similarly, were there any changes on the client (Qlik) side?

@balaji.ramaswamy No there is no change in dremio side, But we upgraded the data gateway in qlik and also 3 weeks ago qlik was having some issue but it was resolved in one day.

@fenilj The fix to Qlik, what was the exact issue and steps to fix? Wondering if that broke something on the server side?

Sorry for late reply, But i don’t know what exactly was the issue, But it was from qlik side for particular region and for data gateway, So we simply updated the gateway to latest gateway and we haven’t faced that problem again.