JDBC Frequently Fail to send message because connection is no longer valid

1

I have a microservice that connect to Dremio using JDBC to retrive data to serve on front-end application.

Actually I get frequently the following error:

java.lang.IllegalArgumentException: Attempted to send a message when connection is no longer valid. null <--> null (user client)
at cdjd.com.google.common.base.Preconditions.checkArgument(Preconditions.java:191)
at cdjd.com.dremio.exec.rpc.RequestIdMap.createNewRpcListener(RequestIdMap.java:90)
at cdjd.com.dremio.exec.rpc.RemoteConnection.createNewRpcListener(RemoteConnection.java:172)
at cdjd.com.dremio.exec.rpc.RpcBus.send(RpcBus.java:161)
at cdjd.com.dremio.exec.rpc.RpcBus.send(RpcBus.java:123)
at cdjd.com.dremio.exec.rpc.BasicClient.send(BasicClient.java:170)
at cdjd.com.dremio.sabot.rpc.user.UserClient.submitQuery(UserClient.java:102)
at cdjd.com.dremio.exec.client.DremioClient.runQuery(DremioClient.java:741)
at com.dremio.jdbc.impl.DremioCursor.loadInitialSchema(DremioCursor.java:571)
at com.dremio.jdbc.impl.DremioResultSetImpl.execute(DremioResultSetImpl.java:1909)
at com.dremio.jdbc.impl.DremioResultSetImpl.execute(DremioResultSetImpl.java:73)
at cdjd.org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:630)
at com.dremio.jdbc.impl.DremioMetaImpl.prepareAndExecute(DremioMetaImpl.java:184)
at cdjd.org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
at com.dremio.jdbc.impl.DremioConnectionImpl.prepareAndExecuteInternal(DremioConnectionImpl.java:123)
at cdjd.org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
at cdjd.org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:218)
at com.dremio.jdbc.impl.DremioStatementImpl.executeQuery(DremioStatementImpl.java:108)
at io.agroal.pool.wrapper.StatementWrapper.executeQuery(StatementWrapper.java:122)
....

I have search on the forum and the problem could be linked to carbage collection of Dremio Worker node ( actually I have only one worker).

I can’t understand why this happen so frequently. Can someone explain how to solve this problem. As I mentioned above it is used to serving data to a front end application and this issue is a little bit noise.

1 Like
2

@stefano.castoldi

Do you get this error when running a query or on connection. Send us the Dremio server.log and let us review it to see if there are any known errors you are running into

3

@balaji.ramaswamy

I get this error on query executions. How Can I send you the logs?

4

Hi @stefano.castoldi,

Do you have any kind of load balancer in front of the Dremio JDBC/ODBC service?

5

Hi,
I know it has been a while since this issue was reported but we currently facing the same one and if Dremio team has any update on this or anyone who has previously experienced this problem found solution I would be grateful for sharing anything helpful.
Thank you

Here is the stacktrace we are getting trying to obtain some metadata:

 Caused by: java.sql.SQLException: Failure getting metadata
        at com.dremio.jdbc.impl.DremioMetaServerImpl$MetadataAdapter.getMeta(DremioMetaServerImpl.java:126)
        at com.dremio.jdbc.impl.DremioMetaServerImpl.getSchemas(DremioMetaServerImpl.java:218)
        at com.dremio.jdbc.impl.DremioMetaImpl.getSchemas(DremioMetaImpl.java:139)
        at cdjd.org.apache.calcite.avatica.AvaticaDatabaseMetaData$10.call(AvaticaDatabaseMetaData.java:696)
        at cdjd.org.apache.calcite.avatica.AvaticaDatabaseMetaData$10.call(AvaticaDatabaseMetaData.java:692)
        at cdjd.org.apache.calcite.avatica.AvaticaConnection.invokeWithRetries(AvaticaConnection.java:793)
        at cdjd.org.apache.calcite.avatica.AvaticaDatabaseMetaData.getSchemas(AvaticaDatabaseMetaData.java:691)
        at com.dremio.jdbc.impl.DremioDatabaseMetaDataImpl.getSchemas(DremioDatabaseMetaDataImpl.java:1586)
        at cdjd.org.apache.calcite.avatica.AvaticaDatabaseMetaData.getSchemas(AvaticaDatabaseMetaData.java:714)
        at com.dremio.jdbc.impl.DremioDatabaseMetaDataImpl.getSchemas(DremioDatabaseMetaDataImpl.java:1151)
       
        ... 67 more
Caused by: cdjd.com.dremio.exec.rpc.RpcException: Failure sending message. Attempted to send a message when connection is no longer valid. null <--> null (user client)
        at cdjd.com.dremio.exec.rpc.RpcBus.send(RpcBus.java:180)
        at cdjd.com.dremio.exec.rpc.RpcBus.send(RpcBus.java:134)
        at cdjd.com.dremio.exec.rpc.RpcBus.send(RpcBus.java:127)
        at cdjd.com.dremio.exec.rpc.BasicClient.send(BasicClient.java:178)
        at cdjd.com.dremio.exec.client.DremioClient.getSchemas(DremioClient.java:601)
        ... 78 more
Caused by: java.lang.IllegalStateException: Attempted to send a message when connection is no longer valid. null <--> null (user client)
        at cdjd.com.google.common.base.Preconditions.checkState(Preconditions.java:590)
        at cdjd.com.dremio.exec.rpc.RequestIdMap.createNewRpcListener(RequestIdMap.java:143)
        at cdjd.com.dremio.exec.rpc.RemoteConnection.createNewRpcListener(RemoteConnection.java:172)
        at cdjd.com.dremio.exec.rpc.RpcBus.send(RpcBus.java:173)
        ... 82 more