dremioDeamon HDP java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration

21

in app master I have:
17/12/08 12:25:36 INFO producer.SyncProducer: Connected to dn-10.fqdn:43501 for producing
17/12/08 12:25:36 INFO producer.SyncProducer: Disconnecting from dn-10.fqdn:43501
17/12/08 12:25:36 INFO network.Processor: Closing socket connection to /10.131.74.110.
17/12/08 12:25:36 INFO producer.SyncProducer: Connected to dn-10.fqdn:43501 for producing
17/12/08 12:25:39 INFO impl.AMRMClientImpl: Received new token for : dn-17.fqdn:45454
17/12/08 12:25:39 INFO impl.ContainerManagementProtocolProxy: Opening proxy : dn-17.fqdn:45454
17/12/08 12:25:42 INFO impl.AMRMClientImpl: Received new token for : dn-2.fqdn:45454
17/12/08 12:25:42 INFO impl.ContainerManagementProtocolProxy: Opening proxy : dn-2.fqdn:45454
17/12/08 12:25:46 INFO impl.AMRMClientImpl: Received new token for : dn-7.fqdn:45454
17/12/08 12:25:46 INFO impl.ContainerManagementProtocolProxy: Opening proxy : dn-7.fqdn:45454
17/12/08 12:25:49 INFO impl.AMRMClientImpl: Received new token for : dn-13.fqdn:45454
17/12/08 12:25:49 INFO impl.ContainerManagementProtocolProxy: Opening proxy : dn-13.fqdn:45454
17/12/08 12:25:52 INFO impl.AMRMClientImpl: Received new token for : dn-9.fqdn:45454
17/12/08 12:25:52 INFO impl.ContainerManagementProtocolProxy: Opening proxy : dn-9.fqdn:45454
17/12/08 12:25:55 INFO impl.AMRMClientImpl: Received new token for : dn-11.fqdn:45454
17/12/08 12:25:55 INFO impl.ContainerManagementProtocolProxy: Opening proxy : dn-11.fqdn:45454
17/12/08 12:25:58 INFO impl.AMRMClientImpl: Received new token for : dn-18.fqdn:45454
17/12/08 12:25:58 INFO impl.ContainerManagementProtocolProxy: Opening proxy : dn-18.fqdn:45454
17/12/08 12:26:01 INFO impl.AMRMClientImpl: Received new token for : dn-12.fqdn:45454
17/12/08 12:26:01 INFO impl.ContainerManagementProtocolProxy: Opening proxy : dn-12.fqdn:45454
17/12/08 12:26:04 INFO impl.AMRMClientImpl: Received new token for : dn-1.fqdn:45454
17/12/08 12:26:04 INFO impl.ContainerManagementProtocolProxy: Opening proxy : dn-1.fqdn:45454
17/12/08 12:26:07 INFO impl.AMRMClientImpl: Received new token for : dn-6.fqdn:45454
17/12/08 12:26:07 INFO impl.ContainerManagementProtocolProxy: Opening proxy : dn-6.fqdn:45454

and in runnableResource I have :
listening for transport dt_socket at address: 40615 (with different port )

22

It feels like you have enabled provisioning debug in dremio.conf:
debug.yarnremote.enabled: true

Please either comment it or set to false. It puts workers into ready for remote debugging state

Please stop Provisioning cluster after that (“Stop” on provisioning UI) and restart your Dremio so property will take effect.

23

work fine, thanks you very much.

So to resume with HDP 2.6.3 and dremio 1.3.0 we need to Copy /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar to $DREMIO_HOME/jars/3rdparty

24

For now - yes. I think the reason being RequestHedgingRMFailoverProxyProvider is enabled somewhere in hdp yarn-site.xml you are sourcing on Dremio classpath and it makes Dremio deal with the class it does not have, otherwise why would it throw java.lang.ClassNotFoundException

25

Yes indeed, I just check in my ambari conf and it set to RequestHedgingRMFailoverProxyProvider

According hdp documentation, the default value is org.apache.hadoop.yarn.client.RMFailoverProxyProvider

Should I change it ? I will lost automatic fail over ?

sources: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_hadoop-high-availability/content/ha-rm-config-failover.html

26

Pirion . no change is required on the HDP end. You need to do the following:

  1. copy the jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar to $DREMIO_HOME/jars/3rdparty

  2. restart the coordinator

  3. Provision the executors using Dremio -> Admin-> Provisioning screen

27

Advice was not to change what Ambari sets. I am not HDP expert though :slight_smile:

28

Hi @Pirion

Did you try steps suggested by @Ajay? Any updates?

Thanks,
@balaji.ramaswamy

29

All is fine now. Thanks

30

Hi @Pirion

What fixed the issue?

Thanks,
@balaji.ramaswamy

31

copy the jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar to $DREMIO_HOME/jars/3rdparty