Intermittent connection to?

Hi,

We just upgraded our cluster from 3.1.11 to 3.3.2, everything look smooth except I noticed there are tons of this error in server.log on the coordinator nodes. This message(s) doesn’t really tell anything to us, which side that went wrong.

2019-09-19 08:36:33,155 [qtp1993004305-254] ERROR c.d.dac.server.socket.SocketServlet - Failure in web socket handling.
java.io.EOFException: Disconnected
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:323) ~[websocket-common-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.websocket.common.WebSocketSession.onClosed(WebSocketSession.java:393) ~[websocket-common-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.io.AbstractConnection.onClosed(AbstractConnection.java:231) [jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.io.AbstractConnection.onClose(AbstractConnection.java:224) [jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.io.SelectorManager.connectionClosed(SelectorManager.java:345) [jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint.run(ManagedSelector.java:958) [jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) [jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) [jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_222]

Any idea?

Hi,

The error is about sockets being closed unexpectedly, which is not a problem since it can happen when users close browsers for example. We have an internal ticket to stop logging websocket exceptions since its noise that isn’t helpful.

1 Like

Thanks for the response.

So this can be safely ignored.
Got it!

Correct, they can be ignored.

Is this error the same or different?

2020-07-13 03:22:06,918 [qtp917950045-269] ERROR c.d.dac.server.socket.SocketServlet - Failure in web socket handling.
org.eclipse.jetty.io.EofException: null
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:283)
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264)
at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:584)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:181)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:511)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
at sun.nio.ch.IOUtil.write(IOUtil.java:148)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:263)
… 22 common frames omitted

@desi

This error looks different, is this happening during startup?

not during start up. it occurs sporadicly much after startup, below error is showing timeout of 300000 ms. i’ve searched long and hard on this error. in the documentation it says something about “web sockets should be setup correctly in your environment” however there is no information beyond that …

2020-07-13 03:54:23,392 [Connector-Scheduler-75cc4e8-1] ERROR c.d.dac.server.socket.SocketServlet - Failure in web socket handling.

org.eclipse.jetty.websocket.api.CloseException: java.util.concurrent.TimeoutException: Idle timeout expired: 300000/300000 ms

at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onReadTimeout(AbstractWebSocketConnection.java:567)

at org.eclipse.jetty.io.AbstractConnection.onFillInterestedFailed(AbstractConnection.java:172)

at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillInterestedFailed(AbstractWebSocketConnection.java:540)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.failed(AbstractConnection.java:317)

at org.eclipse.jetty.io.FillInterest.onFail(FillInterest.java:138)

at org.eclipse.jetty.io.AbstractEndPoint.onIdleExpired(AbstractEndPoint.java:407)

at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)

at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 300000/300000 ms

… 9 common frames omitted

2020-07-13 03:57:47,053 [Connector-Scheduler-75cc4e8-1] ERROR c.d.dac.server.socket.SocketServlet - Failure in web socket handling.

org.eclipse.jetty.websocket.api.CloseException: java.util.concurrent.TimeoutException: Idle timeout expired: 300000/300000 ms

at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onReadTimeout(AbstractWebSocketConnection.java:567)

at org.eclipse.jetty.io.AbstractConnection.onFillInterestedFailed(AbstractConnection.java:172)

at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillInterestedFailed(AbstractWebSocketConnection.java:540)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.failed(AbstractConnection.java:317)

at org.eclipse.jetty.io.FillInterest.onFail(FillInterest.java:138)

at org.eclipse.jetty.io.AbstractEndPoint.onIdleExpired(AbstractEndPoint.java:407)

at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)

at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 300000/300000 ms

… 9 common frames omitted

@doron

The above message can be ignored, other than the message, are you facing any problems?

Thanks
Bali