dremioDeamon HDP java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration

1

Hi,

I just dremio 1.3 and hdfs source work fine.

but when I try to start Yarn provisioning I got an error on yarn :

Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:70)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
… 6 more

Someone can help me ?

2

Hi, did you follow these instructions when setting up Dremio to run on your HDP cluster?

3

yes and it fail on the on the step 3. Monitor and manage YARN executor nodes.

dremioDeamon are launch on yarn but fail with the java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration error

4

Could you provide full stacktrace? Also is AppMaster (should be container 1) launched fine and only other containers have this issue or AppMaster as well?

5

it seem the app master fail too.

log from yarn:

Java HotSpot™ 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:70)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
… 6 more

from dremio :
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Yarn application completed with failure DremioDaemon, da60fd32-c1c2-444b-ba23-0bbd45e55e82.
at twill.com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294) ~[dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at twill.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:281) ~[dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at twill.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.provision.yarn.service.YarnService$OnTerminatingRunnable.run(YarnService.java:397) ~[dremio-yarn-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at org.apache.twill.internal.AbstractExecutionServiceController$4.failed(AbstractExecutionServiceController.java:113) [dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at org.apache.twill.internal.ListenerExecutor$5.run(ListenerExecutor.java:123) [dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at org.apache.twill.common.Threads$1.execute(Threads.java:35) [dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at org.apache.twill.internal.ListenerExecutor.failed(ListenerExecutor.java:119) [dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at org.apache.twill.internal.AbstractExecutionServiceController$ListenerExecutors.failed(AbstractExecutionServiceController.java:282) [dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at twill.com.google.common.util.concurrent.AbstractService$6$1.run(AbstractService.java:444) [dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at org.apache.twill.common.Threads$1.execute(Threads.java:35) [dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at twill.com.google.common.util.concurrent.AbstractService$ListenerExecutorPair.execute(AbstractService.java:470) [dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at twill.com.google.common.util.concurrent.AbstractService$6.run(AbstractService.java:442) [dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at twill.com.google.common.util.concurrent.AbstractService.executeListeners(AbstractService.java:369) [dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at twill.com.google.common.util.concurrent.AbstractService.notifyFailed(AbstractService.java:313) [dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at twill.com.google.common.util.concurrent.AbstractIdleService$1$2.run(AbstractIdleService.java:60) [dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: java.lang.RuntimeException: Yarn application completed with failure DremioDaemon, da60fd32-c1c2-444b-ba23-0bbd45e55e82.
at org.apache.twill.yarn.YarnTwillController.doShutDown(YarnTwillController.java:202) ~[dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at org.apache.twill.internal.AbstractZKServiceController.shutDown(AbstractZKServiceController.java:98) ~[dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at org.apache.twill.internal.AbstractExecutionServiceController$ServiceDelegate.shutDown(AbstractExecutionServiceController.java:192) ~[dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at twill.com.google.common.util.concurrent.AbstractIdleService$1$2.run(AbstractIdleService.java:57) [dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
… 1 common frames omitted

application overview ;
User: dremio
Name: DremioDaemon
Application Type: YARN
Application Tags:
Application Priority: 0 (Higher Integer value indicates higher priority)
YarnApplicationState: FAILED
Queue: dremio
FinalStatus Reported by AM: FAILED
Started: Tue Dec 05 09:49:14 -0500 2017
Elapsed: 2sec
Tracking URL: History
Log Aggregation Status SUCCEEDED
Diagnostics:
Application application_1512474297083_0008 failed 2 times due to AM Container for appattempt_1512474297083_0008_000002 exited with exitCode: 1
For more detailed output, check the application tracking page: http://:8088/cluster/app/application_1512474297083_0008 Then click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e15_1512474297083_0008_02_000001
Exit code: 1
Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Java HotSpot™ 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:70)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
… 6 more
Failing this attempt. Failing the application.
Unmanaged Application: false
Application Node Label expression:
AM container Node Label expression: <DEFAULT_PARTITION>

application metrics:
Total Resource Preempted: <memory:0, vCores:0>
Total Number of Non-AM Containers Preempted: 0
Total Number of AM Containers Preempted: 0
Resource Preempted from Current Attempt: <memory:0, vCores:0>
Number of Non-AM Containers Preempted from Current Attempt: 0
Aggregate Resource Allocation: 2352 MB-seconds, 1 vcore-seconds

6

Do you have logs form AM container or what you posted is the one?
Do you see on hdfs following directory: /user/dremio/DremioDaemon?
It feels like something is wrong with either jar or classpath or both.

7

what i posted is the that.

yes I have some file in /user/dremio/DremioDaemon/da60fd32-c1c2-444b-ba23-0bbd45e55e82

8

could you do “hdfs fs -lsr /user/dremio/DremioDaemon/da60fd32-c1c2-444b-ba23-0bbd45e55e82” ?

9

rw-r–r-- 3 dremio hdfs 2800330 2017-12-05 09:49 /user/dremio/DremioDaemon/da60fd32-c1c2-444b-ba23-0bbd45e55e82/4256de8a1a08b9deefbc0ca63946cb5c-application.53042e87-b3ae-4ef5-a594-a22f7095b069.jar
-rw-r–r-- 3 dremio hdfs 232820148 2017-12-05 09:49 /user/dremio/DremioDaemon/da60fd32-c1c2-444b-ba23-0bbd45e55e82/dremio-daemon-bundle.jar.f303b68f-e265-44ed-b7d7-aa0cd652221f.jar
-rw-r–r-- 3 dremio hdfs 6340 2017-12-05 09:49 /user/dremio/DremioDaemon/da60fd32-c1c2-444b-ba23-0bbd45e55e82/launcher.9cfeff55-8da3-4d8e-8088-8b67d15677ff.jar
-rw-r–r-- 3 dremio hdfs 918 2017-12-05 09:49 /user/dremio/DremioDaemon/da60fd32-c1c2-444b-ba23-0bbd45e55e82/localizeFiles.903c8c32-6150-4c0a-93bd-4bdb7125e085.json
-rw-r–r-- 3 dremio hdfs 3793 2017-12-05 09:49 /user/dremio/DremioDaemon/da60fd32-c1c2-444b-ba23-0bbd45e55e82/logback.xml.78098a21-e640-49f9-ad41-79bc68af0257.xml
-rw-r–r-- 3 dremio hdfs 2806 2017-12-05 09:49 /user/dremio/DremioDaemon/da60fd32-c1c2-444b-ba23-0bbd45e55e82/runtime.config.954b309e-36c6-475f-9fbd-1e1f652678c8.jar
-rw-r–r-- 3 dremio hdfs 17825823 2017-12-05 09:49 /user/dremio/DremioDaemon/da60fd32-c1c2-444b-ba23-0bbd45e55e82/twill.88605cf0-4c9c-4ffe-a66e-e385121cad69.jar

10

It really feels like containers are not picking up classpath of local hadoop installation. Are you able to run other YARN apps on this cluster where jar you deploy for the application does not include hadoop jar(s)?
I am not very familiar with HDP installation

11

I launched from dremio host a yarn application and it finished successfully:

yarn org.apache.hadoop.yarn.applications.distributedshell.Client -shell_command ls -num_containers 3 -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -timeout 300000 --queue dremio

12

Can you check following in your yarn-site.xml on Node Manager nodes:

<property>
  <name>yarn.application.classpath</name>
  <value>$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*</value>
</property>

or something along those lines.

13

I check my yarn-site.xml on hadoop cluster and dremio host.
it’s pretty much the same :

<property>
<name>yarn.application.classpath</name>
<value>/etc/hadoop/conf,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*,/usr/hdp/current/ext/hadoop/*</value>
</property>

14

Hmmm… I am kind of out of ideas at this point. Is there is a way to get info about that AppMaster container - I think HDP gathers content of the container in terms of jar as well as shell script that starts container?

15

I make fresh install of dremio after resync conf on host with ambari.

I have now an other error:
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2248) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:185) ~[hadoop-yarn-common-2.8.0.jar:na]
at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:118) ~[hadoop-yarn-common-2.8.0.jar:na]
at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:93) ~[hadoop-yarn-common-2.8.0.jar:na]
at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72) ~[hadoop-yarn-common-2.8.0.jar:na]
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:210) ~[hadoop-yarn-client-2.8.0.jar:na]
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) ~[hadoop-common-2.8.0.jar:na]
at org.apache.twill.internal.yarn.Hadoop21YarnAppClient.createYarnClient(Hadoop21YarnAppClient.java:72) ~[dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at org.apache.twill.internal.yarn.Hadoop21YarnAppClient.createLauncher(Hadoop21YarnAppClient.java:79) ~[dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at org.apache.twill.yarn.YarnTwillPreparer.start(YarnTwillPreparer.java:368) [dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at org.apache.twill.yarn.YarnTwillPreparer.start(YarnTwillPreparer.java:361) [dremio-twill-shaded-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.provision.yarn.YarnController.startCluster(YarnController.java:100) [dremio-yarn-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.provision.yarn.service.YarnService.startClusterAsync(YarnService.java:131) [dremio-yarn-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.provision.yarn.service.YarnService.startCluster(YarnService.java:111) [dremio-yarn-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.provision.service.ProvisioningServiceImpl.startCluster(ProvisioningServiceImpl.java:283) [dremio-provision-common-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.provision.service.ProvisioningServiceImpl.createCluster(ProvisioningServiceImpl.java:148) [dremio-provision-common-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at com.dremio.provision.resource.ProvisioningResource.createCluster(ProvisioningResource.java:141) [dremio-provision-common-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at com.dremio.dac.server.AccessLogFilter.doFilter(AccessLogFilter.java:119) [dremio-dac-backend-1.3.0-201711211846350824-31f8d91.jar:1.3.0-201711211846350824-31f8d91]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2216) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2240) ~[hadoop-common-2.8.0.jar:na]
… 64 common frames omitted
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2122) ~[hadoop-common-2.8.0.jar:na]
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2214) ~[hadoop-common-2.8.0.jar:na]
… 65 common frames omitted

16

Yes, HDP pulls a lot of changes from upstream, so hadoop jars included in Dremio distro are not up to hadoop 3.0 level yet
Could you do following:
Copy /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar to $DREMIO_HOME/jars/3rdparty

on coordinator node

17

It’s what I do :wink:
now the Dremio start without error but workers stay in Provisioning or Disconnected

in dremio log I have:
017-12-08 12:26:01,150 [ApplicationMasterService] INFO o.a.t.i.a.RunnableProcessLauncher - [dn-10.fqdn] RunnableProcessLauncher:doLaunch(RunnableProcessLauncher.java:71) - Launching in container container_e18_1512734340085_0043_01_000009 at dn-12.fqdn:45454, [TWILL_DEBUG_PORT=$($JAVA_HOME/bin/java -cp launcher.jar org.apache.twill.launcher.FindFreePort) && $JAVA_HOME/bin/java -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=$TWILL_DEBUG_PORT -Dtwill.debug.port=$TWILL_DEBUG_PORT -Djava.io.tmpdir=tmp -Dyarn.container=$YARN_CONTAINER_ID -Dtwill.runnable=$TWILL_APP_NAME.$TWILL_RUNNABLE_NAME -cp launcher.jar:$HADOOP_CONF_DIR -Xmx4096m -Dpaths.dist=hdfs://ns1/data/dremio/pdfs -Dmaster.name=dremio.fqdn-Dzookeeper=qn1:2181,qn2:2181,qn3:2181 -Dpaths.spilling=[“file:///var/log/dremio”] -Ddebug.autoPort=true -Dmaster.port=45678 -Ddremio.classpath.scanning.cache.enabled=false -Dpaths.local=/tmp/dremio -Dservices.coordinator.enabled=false -Dlogback.configurationFile=logback.xml -XX:MaxPermSize=512m -XX:MaxDirectMemorySize=12288m -Dservices.coordinator.enabled=false org.apache.twill.launcher.TwillLauncher org.apache.twill.internal.container.TwillContainerMain true 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr]

no error in dremio log.
in yarn log I have:
12:26:07.212 [ApplicationMasterService] INFO o.a.t.i.a.ApplicationMasterService - Runnable BundledJarRunnable fully provisioned with 10 instances.

18

Can you check from RM UI if DremioDaemon job is running and look at AppMaster logs for the job and if some containers are started - look at their logs

19

DremioDaemon is running and have
all 10 are Running in RM UI and I have:
1 appMasterResources
10 runnableResources

20

OK, since you are saying that on Dremio UI in provisioning you see workers in “Provisioning” or “Disconnected” and not in “Running” state please look at the logs of some of those 10 containers (you call them runnable resources) if there is any clue there.