Channel shutdown invoked

1

Hi,
I am unable to list the jobs. Please find the attached error and screenshot.
Dremio version 4.5 is running on GKE with 1 master pod ( 32GB Heap, 50 GB Direct memory) & 2 executor pods.
What can be done to avoid this issue?

2020-09-15 04:56:07,061 [qtp131223023-21572] ERROR c.d.d.server.GenericExceptionMapper - Unexpected exception when processing GET http://IP:9047/apiv2/jobs/?sort=st&order=DESCENDING&filter=(qt%3D%3D"UI"%2Cqt%3D%3D"EXTERNAL") : io.grpc.StatusRuntimeException: UNAVAILABLE: Channel shutdown invoked
io.grpc.StatusRuntimeException: UNAVAILABLE: Channel shutdown invoked
at io.grpc.Status.asRuntimeException(Status.java:533)
at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:584)
at java.util.Iterator.forEachRemaining(Iterator.java:115)
at com.dremio.service.jobs.HybridJobsService.searchJobs(HybridJobsService.java:186)
at com.dremio.dac.resource.JobsResource.getJobs(JobsResource.java:99)
at sun.reflect.GeneratedMethodAccessor764.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
at com.dremio.dac.server.tracing.SpanFinishingFilter.doFilter(SpanFinishingFilter.java:46)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
at com.dremio.dac.server.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:52)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)

Screen Shot 2020-09-15 at 10.27.34 AM
Screen Shot 2020-09-15 at 10.27.34 AM1910×992 27.7 KB

2

Hello @unni,

Do you only get this for the Jobs page? Are there any requests to the Jobs page that are successful?

3

@unni, can you supply the coordinator’s server.log (usually under /var/log/dremio) from around the time of the error (2020-09-15 04:56:07)? This may be an example of an bug addressed in a later version of Dremio, but I’d need more information to make that determination.

4

Hello again @unni,

This is indeed a known bug that we have fixed in Dremio 4.7+. Please upgrade your Dremio instance and report back here if this addresses the issue.

5

Thank you for the confirmation.

6

Hi,
We are facing this issue UNAVAILABLE: Channel shutdown invoked while creating the Reflections on the Data Sets. We are using Dremio AWS Edition version 21.2.0 (Community Edition)

Following are the logs when we tried to create reflections manually:

2022-08-03 11:47:14,822 [dremio-general-214] INFO c.d.s.reflection.ReflectionManager - reflection 34a601d9-a469-4e5b-9796-407043f76268 (Raw_dl_quickaction) is due for refresh
2022-08-03 11:47:14,920 [dremio-general-214] WARN c.d.s.reflection.ReflectionManager - failed to refresh reflection 34a601d9-a469-4e5b-9796-407043f76268 (Raw_dl_quickaction)
io.grpc.StatusRuntimeException: UNAVAILABLE: Channel shutdown invoked
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
at com.dremio.services.nessie.grpc.api.TreeServiceGrpc$TreeServiceBlockingStub.getDefaultBranch(TreeServiceGrpc.java:708)
at com.dremio.services.nessie.grpc.client.v1api.GrpcApiV1Impl.lambda$getDefaultBranch$0(GrpcApiV1Impl.java:100)
at com.dremio.services.nessie.grpc.client.GrpcExceptionMapper.handleNessieNotFoundEx(GrpcExceptionMapper.java:168)
at com.dremio.services.nessie.grpc.client.v1api.GrpcApiV1Impl.getDefaultBranch(GrpcApiV1Impl.java:97)
at com.dremio.plugins.NessieClientImpl.getDefaultBranch(NessieClientImpl.java:70)
at com.dremio.exec.store.iceberg.nessie.IcebergNessieTableOperations.getDefaultBranch(IcebergNessieTableOperations.java:129)
at com.dremio.exec.store.iceberg.nessie.IcebergNessieTableOperations.doRefresh(IcebergNessieTableOperations.java:73)
at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:95)
at org.apache.iceberg.BaseTable.refresh(BaseTable.java:59)
at com.dremio.exec.store.iceberg.model.IcebergBaseCommand.loadTable(IcebergBaseCommand.java:366)
at com.dremio.exec.store.iceberg.model.IcebergBaseModel.getIcebergTable(IcebergBaseModel.java:151)
at com.dremio.service.reflection.ReflectionManager.getIcebergTable(ReflectionManager.java:1108)
at com.dremio.service.reflection.ReflectionManager.getIcebergSnapshot(ReflectionManager.java:1054)
at com.dremio.service.reflection.ReflectionManager.startRefresh(ReflectionManager.java:1023)
at com.dremio.service.reflection.ReflectionManager.handleEntry(ReflectionManager.java:404)
at com.dremio.service.reflection.ReflectionManager.handleEntries(ReflectionManager.java:341)
at com.dremio.service.reflection.ReflectionManager.sync(ReflectionManager.java:223)
at com.dremio.service.reflection.ReflectionManager.run(ReflectionManager.java:202)
at com.dremio.common.WakeupHandler$1.run(WakeupHandler.java:67)
at com.dremio.context.RequestContext.run(RequestContext.java:95)
at com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$3(ContextMigratingExecutorService.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
.

7

@AkshayJindal Engineering is actively working on a fix for this, will be part of a 21.x sub release