After we migrated our dremio 19.5 instance to a new aks cluster, we receive many of this (and other) errors in the logs:
2022-09-16 07:09:36,226 [FABRIC-8] INFO c.d.s.fabric.EnterpriseFabricServer - [FABRIC]: Channel closed /10.18.108.53:45678 ↔ /10.18.100.247:47514 (fabric server)
2022-09-16 07:09:36,226 [FABRIC-8] INFO com.dremio.exec.rpc.MessageDecoder - Channel is closed, discarding remaining 168 byte(s) in buffer.
2022-09-16 07:10:06,227 [FABRIC-9] ERROR c.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication. Connection: /10.18.108.53:45678 ↔ /10.18.100.247:53776 (fabric server). Closing connection.
io.netty.handler.codec.CorruptedFrameException: Expected to read a tag of 10 but actually received a value of 69. Happened after reading 0 message.
at com.dremio.exec.rpc.MessageDecoder.checkTag(MessageDecoder.java:223)
at com.dremio.exec.rpc.MessageDecoder.decodeMessage(MessageDecoder.java:143)
at com.dremio.exec.rpc.MessageDecoder.decode(MessageDecoder.java:69)
at com.dremio.services.fabric.FabricProtobufLengthDecoder.decode(FabricProtobufLengthDecoder.java:37)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:750)
What can we do?