Concurrent test for arrow flight cause Timeout error frequently

Dremio University
1

Hello everyone,

I run arrow flight concurrent test according to java ExecutorService and the concurrent number is 1000. Around more than 400 request will occurs timeout error as below info. I want to know if it is dremio server timeout. If not. As checked, the timeout default value is 30s. How can I config my service timeout value? Thanks in advance.

org.apache.arrow.flight.FlightRuntimeException: UNAVAILABLE: io exception
at org.apache.arrow.flight.CallStatus.toRuntimeException(CallStatus.java:131)
at org.apache.arrow.flight.grpc.StatusUtils.fromGrpcRuntimeException(StatusUtils.java:152)
at org.apache.arrow.flight.auth.ClientAuthWrapper.doClientAuth(ClientAuthWrapper.java:56)
at org.apache.arrow.flight.FlightClient.authenticate(FlightClient.java:183)
at org.apache.arrow.flight.FlightClient.authenticateBasic(FlightClient.java:172)
at com.scb.fmdpf.dqsl.client.ArrowFlightConnectDremioPerformanceTest.flight_dremio_connecting_test(ArrowFlightConnectDremioPerformanceTest.java:240)
at com.scb.fmdpf.dqsl.client.ArrowFlightConnectDremioPerformanceTest.flight_BCDF_simple_35Fields_test_1k(ArrowFlightConnectDremioPerformanceTest.java:68)
at com.scb.fmdpf.dqsl.client.ArrowFlightConnectDremioPerformanceTest$1.run(ArrowFlightConnectDremioPerformanceTest.java:187)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: xxxxx.com/xxxxxxxx:32010
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:263)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:515)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
… 1 more

2

Hi ,
Did you tried the .ArrowFlightConnectDremioPerformanceTest with creating the flight client for each request or reusing the flight client and making concurrent grpc calls?. Because we are not able to make concurrent grpc calls using same arrow flight client.
Exception:
jd.org.apache.arrow.flight.FlightRuntimeException: INTERNAL:
at cfjd.org.apache.arrow.flight.CallStatus.toRuntimeException(CallStatus.java:131)
at cfjd.org.apache.arrow.flight.grpc.StatusUtils.fromGrpcRuntimeException(StatusUtils.java:164)
at cfjd.org.apache.arrow.flight.grpc.StatusUtils.fromThrowable(StatusUtils.java:185)
at cfjd.org.apache.arrow.flight.FlightStream$Observer.onError(FlightStream.java:454)
at cfjd.org.apache.arrow.flight.FlightClient$1.onError(FlightClient.java:350)
at cfjd.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:491)
at cfjd.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at cfjd.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at cfjd.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at cfjd.org.apache.arrow.flight.grpc.ClientInterceptorAdapter$FlightClientCallListener.onClose(ClientInterceptorAdapter.java:117)
at cfjd.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
at cfjd.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
at cfjd.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
at cfjd.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
at cfjd.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at cfjd.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Expand (20 lines)

Thanks,
Pavan