Unable to connect to HDFS

1

I am unable to connect to HDFS (localhost); previously, I use to do this in version 1.0.8 …

I am just testing a simple case; the hadoop is on localhost and the port is 8020. I tried both with/without impersonation… nothing worked.

INFO  c.d.e.s.StoragePluginRegistryImpl - User Error Occurred [ErrorId: 45754499-a4cb-4e5f-97e6-cbba6b5b0d92]   com.dremio.common.exceptions.UserException: Unable to connect to source localhost_hdfs  

    at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:648) ~[dremio-common-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
    at com.dremio.exec.store.StoragePluginRegistryImpl.create(StoragePluginRegistryImpl.java:465) [dremio-sabot-kernel-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
    at com.dremio.exec.store.StoragePluginRegistryImpl.createOrUpdate(StoragePluginRegistryImpl.java:271) [dremio-sabot-kernel-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
    at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:125) [dremio-dac-backend-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
    at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:117) [dremio-dac-backend-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
    at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:81) [dremio-dac-backend-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_102]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_102]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_102]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_102]
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
   at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
2

Hi @HLNA,

If you are trying to add HDFS as a DS in Dremio, I’ve seen it that in some cases you need to add a hostname entry in /etc/hosts: I’ve had instances before when I wasn’t able to add HDFS and Hive DS, since the hostname wasn’t resolved properly.

Also make sure that:
/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar file is copied over to $DREMIO_HOME/jars/3rdparty dir (on Dremio coordinator).

Let me know if this worked for you.

Thanks,
Danny

3

Hi @dbrody Thanks for your help! I went through the logs and it seems the problem was with datanode issues; Did complete reset and was able to connect;

4

Excellent @HLNA, I’m glad you are now able to connect to your HDFS datasource, have a great day!

5

Thx for wonderful follow-ups and support @dbrody

6

You are welcome @HLNA, have a great rest of your day! :slight_smile:

7

I am facing the same issue. I ensured the Jar is available in third party folder.I am running Dremio by provisioning using yarn

MY Name node and data node is running

This is what I provide
Name node : namenode hostname
port :8020

Got this error in Admin UI
There was an issue communicating with Dremio.
show less
Your Internet connection may be offline, or WebSockets to Dremio are being blocked.

8

Can you ensure there are no issues with the Hadoop namenode service and that there is open communication between the 2 boxes? Also are there any more details in server.log?