Dremio Yarn Kerberos Errors

yufeldman · April 26, 2018, 7:22pm

Do you get the above exception upon:

Trying to start up Provisioning (Dremio coordinator server.log)
Spinning up AppMaster (RM UI Logs for Dremio YARN application container 1)
Spinning up Dremio container(s) (RM UI Logs for Dremio YARN application any container except container 1)

Zack · April 26, 2018, 7:31pm

Error presented itself after failing to provision Yarn container via UI.
server.out reports no errors

server.log

2018-04-26 19:23:51,764 [Curator-ServiceCache-0] INFO c.d.d.s.exec.MasterStatusListener - New master node DREMIO_HOST_COORD:45678 registered itself.
2018-04-26 19:23:52,330 [main] INFO c.d.d.s.TimingApplicationEventListener - ApplicationEventListener.onEvent INITIALIZATION_START
2018-04-26 19:23:53,338 [main] INFO c.d.d.s.TimingApplicationEventListener - ApplicationEventListener.onEvent INITIALIZATION_APP_FINISHED
2018-04-26 19:23:53,340 [main] INFO c.d.d.s.TimingApplicationEventListener - ApplicationEventListener.onEvent INITIALIZATION_FINISHED
2018-04-26 19:23:53,448 [main] INFO com.dremio.dac.server.WebServer - Started on http://localhost:9047
2018-04-26 19:24:17,364 [251dd89f-0bab-76c7-e0a2-c9a28cff3e00:foreman] WARN c.d.exec.planner.logical.RexToExpr - Converting exact decimal into approximate decimal. Should be fixed once decimal is implemented.
2018-04-26 19:24:17,403 [251dd89f-0bab-76c7-e0a2-c9a28cff3e00:foreman] INFO c.d.e.p.s.h.c.ExecutionPlanCreator - User Error Occurred [ErrorId: 280e355e-3c15-4583-9175-622dac4a4f6b]
com.dremio.common.exceptions.UserException: No executors currently available.
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.exec.planner.sql.handlers.commands.ExecutionPlanCreator.getExecutionPlan(ExecutionPlanCreator.java:58) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:68) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:290) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
2018-04-26 19:24:17,451 [251dd89f-0bab-76c7-e0a2-c9a28cff3e00:foreman] INFO query.logger - {“queryId”:“251dd89f-0bab-76c7-e0a2-c9a28cff3e00”,“schema”:“sys”,“queryText”:“select\n ‘green’ as status,\n nodes.hostname name,\n nodes.ip_address ip,\n nodes.fabric_port port,\n cpu cpu,\n memory memory \nfrom\n sys.nodes,\n (select\n hostname,\n fabric_port,\n sum(cast(cpuTime as float) / cores) cpu \n from\n sys.threads \n group by\n hostname,\n fabric_port) cpu,\n (select\n hostname,\n fabric_port,\n direct_current * 100.0 / direct_max as memory \n from\n sys.memory) memory \nwhere\n nodes.hostname = cpu.hostname \n and nodes.fabric_port = cpu.fabric_port \n and nodes.hostname = memory.hostname \n and nodes.fabric_port = memory.fabric_port \norder by\n name,\n port”,“start”:1524770656180,“finish”:1524770657404,“outcome”:“FAILED”,“username”:“dremio”,“commandDescription”:“”}

Thanks

yufeldman · April 26, 2018, 7:58pm

The only exception I see is following:

Which means that no Executors are running

Could you please go to RM UI and look if there is DremioDaemon application is running (showing up) in Applications tab on RM UI

If it is - please drill down to it’s logs to see if there is something more meaningful

yufeldman · April 26, 2018, 8:12pm

One more - you may want to link/make sure yarn-site.xml is on Dremio Coordinator classpath (if not yet) - as provisioning needs to contact RM.

Zack · April 26, 2018, 8:28pm

Thanks for sticking with me yufeldman.

It appears I am not actually hitting the RM server. I was not checking the datetime associated with the Yarn logs. Current config without yarn-site.xml fails to hit RM. Now, I added yarn-site.xml back into conf and I am getting errors in server.log. However, I am still not hitting RM since latest datetime is two days back.

To hit RM previously I had to move yarn jar files out and link my own version from my classpath. Your versions tend to be v2.8 and we are using mostly using v2.7 still as part of Horton Works version I listed previously.

server.log

2018-04-26 20:16:10,578 [251dcc75-15e5-fe7c-2f01-d34339f49d00:foreman] INFO c.d.e.p.s.h.c.ExecutionPlanCreator - User Error Occurred [ErrorId: e2ee3eef-1740-4ece-be32-85cb30a582bb]
com.dremio.common.exceptions.UserException: No executors currently available.
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.exec.planner.sql.handlers.commands.ExecutionPlanCreator.getExecutionPlan(ExecutionPlanCreator.java:58) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:68) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:290) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
2018-04-26 20:16:10,579 [251dcc75-15e5-fe7c-2f01-d34339f49d00:foreman] INFO query.logger - {“queryId”:“251dcc75-15e5-fe7c-2f01-d34339f49d00”,“schema”:“sys”,“queryText”:“select\n ‘green’ as status,\n nodes.hostname name,\n nodes.ip_address ip,\n nodes.fabric_port port,\n cpu cpu,\n memory memory \nfrom\n sys.nodes,\n (select\n hostname,\n fabric_port,\n sum(cast(cpuTime as float) / cores) cpu \n from\n sys.threads \n group by\n hostname,\n fabric_port) cpu,\n (select\n hostname,\n fabric_port,\n direct_current * 100.0 / direct_max as memory \n from\n sys.memory) memory \nwhere\n nodes.hostname = cpu.hostname \n and nodes.fabric_port = cpu.fabric_port \n and nodes.hostname = memory.hostname \n and nodes.fabric_port = memory.fabric_port \norder by\n name,\n port”,“start”:1524773770365,“finish”:1524773770578,“outcome”:“FAILED”,“username”:“dremio”,“commandDescription”:“”}
2018-04-26 20:17:00,549 [qtp1982794379-190] ERROR o.a.twill.yarn.YarnTwillPreparer - Failed to submit application DremioDaemon
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2248) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:185) ~[hadoop-yarn-common-2.8.0.jar:na]
at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:118) ~[hadoop-yarn-common-2.8.0.jar:na]
at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:93) ~[hadoop-yarn-common-2.8.0.jar:na]
at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72) ~[hadoop-yarn-common-2.8.0.jar:na]
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:210) ~[hadoop-yarn-client-2.8.0.jar:na]
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) ~[hadoop-common-2.8.0.jar:na]
at org.apache.twill.internal.yarn.Hadoop21YarnAppClient.createYarnClient(Hadoop21YarnAppClient.java:72) ~[dremio-twill-shaded-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at org.apache.twill.internal.yarn.Hadoop21YarnAppClient.createLauncher(Hadoop21YarnAppClient.java:79) ~[dremio-twill-shaded-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at org.apache.twill.yarn.YarnTwillPreparer.start(YarnTwillPreparer.java:368) [dremio-twill-shaded-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at org.apache.twill.yarn.YarnTwillPreparer.start(YarnTwillPreparer.java:361) [dremio-twill-shaded-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.provision.yarn.YarnController.startCluster(YarnController.java:101) [dremio-yarn-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.provision.yarn.service.YarnService.startClusterAsync(YarnService.java:131) [dremio-yarn-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.provision.yarn.service.YarnService.startCluster(YarnService.java:111) [dremio-yarn-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.provision.service.ProvisioningServiceImpl.startCluster(ProvisioningServiceImpl.java:283) [dremio-provision-common-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.provision.service.ProvisioningServiceImpl.createCluster(ProvisioningServiceImpl.java:148) [dremio-provision-common-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.provision.resource.ProvisioningResource.createCluster(ProvisioningResource.java:142) [dremio-provision-common-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_161]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2216) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2240) ~[hadoop-common-2.8.0.jar:na]
… 62 common frames omitted
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2122) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2214) ~[hadoop-common-2.8.0.jar:na]
… 63 common frames omitted
2018-04-26 20:17:00,552 [qtp1982794379-190] ERROR c.d.d.server.GenericExceptionMapper - Unexpected exception when processing POST http://DREMIO_COORD_HOST:9047/apiv2/provision/cluster : java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2248) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:185) ~[hadoop-yarn-common-2.8.0.jar:na]
at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:118) ~[hadoop-yarn-common-2.8.0.jar:na]
at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:93) ~[hadoop-yarn-common-2.8.0.jar:na]
at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72) ~[hadoop-yarn-common-2.8.0.jar:na]
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:210) ~[hadoop-yarn-client-2.8.0.jar:na]
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) ~[hadoop-common-2.8.0.jar:na]
at org.apache.twill.internal.yarn.Hadoop21YarnAppClient.createYarnClient(Hadoop21YarnAppClient.java:72) ~[dremio-twill-shaded-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at org.apache.twill.internal.yarn.Hadoop21YarnAppClient.createLauncher(Hadoop21YarnAppClient.java:79) ~[dremio-twill-shaded-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at org.apache.twill.yarn.YarnTwillPreparer.start(YarnTwillPreparer.java:368) ~[dremio-twill-shaded-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at org.apache.twill.yarn.YarnTwillPreparer.start(YarnTwillPreparer.java:361) ~[dremio-twill-shaded-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.provision.yarn.YarnController.startCluster(YarnController.java:101) ~[dremio-yarn-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.provision.yarn.service.YarnService.startClusterAsync(YarnService.java:131) ~[dremio-yarn-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.provision.yarn.service.YarnService.startCluster(YarnService.java:111) ~[dremio-yarn-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.provision.service.ProvisioningServiceImpl.startCluster(ProvisioningServiceImpl.java:283) ~[dremio-provision-common-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.provision.service.ProvisioningServiceImpl.createCluster(ProvisioningServiceImpl.java:148) ~[dremio-provision-common-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at com.dremio.provision.resource.ProvisioningResource.createCluster(ProvisioningResource.java:142) ~[dremio-provision-common-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_161]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2216) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2240) ~[hadoop-common-2.8.0.jar:na]
… 62 common frames omitted
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2122) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2214) ~[hadoop-common-2.8.0.jar:na]
… 63 common frames omitted

yufeldman · April 26, 2018, 8:37pm

The problem is not that we have hadoop version 2.8 and HW 2.7. HW actually backports JIRAs from hadoop 3.0 (e.g. RequestHedgingRMFailoverProxyProvider - class is from Hadoop 3.x)

Could you please do following:
copy yarn-site.xml to dremio conf directory and from copied version of yarn-site.xml remove property that refers to this class: RequestHedgingRMFailoverProxyProvider

Zack · April 26, 2018, 8:56pm

yufeldman,

Much appreciated support. Yarn container kicked off with no errors. The rest is adding hive and hadoop datasoruces. I will play with that for a time and start a new thread if I hit wall.

Thank you to everyone who chimed in on this post

Topic		Replies	Views
Connecting local Dremio to secured Hadoop cluster	2	1140	May 30, 2018
How to connect dremio to HDFS cluster when Kerberos is installed	18	3656	August 16, 2018
Hortonworks HDFS on Secure Cluster	2	1218	May 7, 2018
Kerberized HDP: HDFS works, Hive doesn't	14	5067	February 17, 2020
Info: Dremio on EMR - Failed to set up YARN	1	1115	April 29, 2019

Dremio Yarn Kerberos Errors

Related Topics