Intermittent connection to?

chafidz · September 19, 2019, 8:48am

Hi,

We just upgraded our cluster from 3.1.11 to 3.3.2, everything look smooth except I noticed there are tons of this error in server.log on the coordinator nodes. This message(s) doesn’t really tell anything to us, which side that went wrong.

2019-09-19 08:36:33,155 [qtp1993004305-254] ERROR c.d.dac.server.socket.SocketServlet - Failure in web socket handling.
java.io.EOFException: Disconnected
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:323) ~[websocket-common-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.websocket.common.WebSocketSession.onClosed(WebSocketSession.java:393) ~[websocket-common-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.io.AbstractConnection.onClosed(AbstractConnection.java:231) [jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.io.AbstractConnection.onClose(AbstractConnection.java:224) [jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.io.SelectorManager.connectionClosed(SelectorManager.java:345) [jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint.run(ManagedSelector.java:958) [jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) [jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) [jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_222]

Any idea?

doron · September 19, 2019, 3:06pm

Hi,

The error is about sockets being closed unexpectedly, which is not a problem since it can happen when users close browsers for example. We have an internal ticket to stop logging websocket exceptions since its noise that isn’t helpful.

chafidz · September 19, 2019, 11:53pm

Thanks for the response.

So this can be safely ignored.
Got it!

doron · September 20, 2019, 4:46am

Correct, they can be ignored.

desi · July 13, 2020, 4:00am

Is this error the same or different?

2020-07-13 03:22:06,918 [qtp917950045-269] ERROR c.d.dac.server.socket.SocketServlet - Failure in web socket handling.
org.eclipse.jetty.io.EofException: null
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:283)
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264)
at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:584)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:181)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:511)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
at sun.nio.ch.IOUtil.write(IOUtil.java:148)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:263)
… 22 common frames omitted

balaji.ramaswamy · July 13, 2020, 4:01am

@desi

This error looks different, is this happening during startup?

desi · July 13, 2020, 4:10am

not during start up. it occurs sporadicly much after startup, below error is showing timeout of 300000 ms. i’ve searched long and hard on this error. in the documentation it says something about “web sockets should be setup correctly in your environment” however there is no information beyond that …

2020-07-13 03:54:23,392 [Connector-Scheduler-75cc4e8-1] ERROR c.d.dac.server.socket.SocketServlet - Failure in web socket handling.

org.eclipse.jetty.websocket.api.CloseException: java.util.concurrent.TimeoutException: Idle timeout expired: 300000/300000 ms

at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onReadTimeout(AbstractWebSocketConnection.java:567)

at org.eclipse.jetty.io.AbstractConnection.onFillInterestedFailed(AbstractConnection.java:172)

at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillInterestedFailed(AbstractWebSocketConnection.java:540)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.failed(AbstractConnection.java:317)

at org.eclipse.jetty.io.FillInterest.onFail(FillInterest.java:138)

at org.eclipse.jetty.io.AbstractEndPoint.onIdleExpired(AbstractEndPoint.java:407)

at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)

at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 300000/300000 ms

… 9 common frames omitted

2020-07-13 03:57:47,053 [Connector-Scheduler-75cc4e8-1] ERROR c.d.dac.server.socket.SocketServlet - Failure in web socket handling.