Unable to add HDFS nor HIVE to DS

isSokOl · June 13, 2018, 7:45am

Hello Dremio Community,

I am using Hadoop 2.9.1 on AWS and I got Dremio up and running. Now trying to connect HDFS and HIVE to Dremio. I opened the three ports 9047 (Dremio), 8020 (HDFS) and 9083 (HIVE), however using
telnet IP 8020
telnet IP 9083
I get that it couldn’t connect to host.

Trying to add to Data Source I got two different errors.

On HDFS I get the message:
Failure while configuring source [HDFS]. Info: Unavailable: Call From my_IP/127.0.0.1 to some_IP:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: ConnectionRefused - HADOOP2 - Apache Software Foundation

On HIVE I get:
Failure while configuring source [HIVE].

These are the two entries I found in /var/log/dremio/access.log:

87.165.145.126 - - [13/Jun/2018:07:25:25 +0000] "PUT /apiv2/source/HIVE?nocache=1528874723273 HTTP/1.1" 400 85 "http://some_IP:9047/source/Samples" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0"
87.165.145.126 - - [13/Jun/2018:07:29:37 +0000] "PUT /apiv2/source/HDFS?nocache=1528874978395 HTTP/1.1" 400 314 "http://some_IP:9047/source/Samples" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0"

Here are the according entries in /var/log/dremio/server.log:

2018-06-13 07:25:25,522 [qtp283748905-106] INFO c.d.exec.catalog.CatalogServiceImpl - User Error Occurred [ErrorId: a90d77ae-8023-44ff-a520-9ecdc03e0e68]
com.dremio.common.exceptions.UserException: Failure while configuring source [HIVE]
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.CatalogServiceImpl.createOrUpdateSource(CatalogServiceImpl.java:629) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:376) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.CatalogServiceImpl.access$600(CatalogServiceImpl.java:100) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.CatalogServiceImpl$SourceModifier.createSource(CatalogServiceImpl.java:946) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.CatalogImpl.createSource(CatalogImpl.java:529) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.DelegatingCatalog.createSource(DelegatingCatalog.java:182) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:147) [dremio-dac-backend-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:138) [dremio-dac-backend-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:134) [dremio-dac-backend-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:80) [dremio-dac-backend-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:420)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:236)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:181)
at com.dremio.exec.store.hive.HiveClient$1.run(HiveClient.java:144)
at com.dremio.exec.store.hive.HiveClient$1.run(HiveClient.java:141)
at com.dremio.exec.store.hive.HiveClient$9.run(HiveClient.java:316)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at com.dremio.exec.store.hive.HiveClient.doAsCommand(HiveClient.java:313)
at com.dremio.exec.store.hive.HiveClient.connect(HiveClient.java:140)
at com.dremio.exec.store.hive.HiveClient.createClient(HiveClient.java:126)
at com.dremio.exec.store.hive.HiveStoragePlugin.start(HiveStoragePlugin.java:573)
at com.dremio.exec.catalog.ManagedStoragePlugin$2.run(ManagedStoragePlugin.java:238)
at com.dremio.concurrent.RenamingRunnable.run(RenamingRunnable.java:36)
at com.dremio.concurrent.SingletonRunnable.run(SingletonRunnable.java:41)
at com.dremio.concurrent.SafeRunnable.run(SafeRunnable.java:40)
at com.dremio.concurrent.Runnables$1.run(Runnables.java:45)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
… 18 more
)
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:272) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:107) ~[guava-18.0.jar:na]
at com.dremio.exec.catalog.CatalogServiceImpl.createOrUpdateSource(CatalogServiceImpl.java:617) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
… 55 common frames omitted
Caused by: java.lang.RuntimeException: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:420)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:236)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:181)
at com.dremio.exec.store.hive.HiveClient$1.run(HiveClient.java:144)
at com.dremio.exec.store.hive.HiveClient$1.run(HiveClient.java:141)
at com.dremio.exec.store.hive.HiveClient$9.run(HiveClient.java:316)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at com.dremio.exec.store.hive.HiveClient.doAsCommand(HiveClient.java:313)
at com.dremio.exec.store.hive.HiveClient.connect(HiveClient.java:140)
at com.dremio.exec.store.hive.HiveClient.createClient(HiveClient.java:126)
at com.dremio.exec.store.hive.HiveStoragePlugin.start(HiveStoragePlugin.java:573)
at com.dremio.exec.catalog.ManagedStoragePlugin$2.run(ManagedStoragePlugin.java:238)
at com.dremio.concurrent.RenamingRunnable.run(RenamingRunnable.java:36)
at com.dremio.concurrent.SingletonRunnable.run(SingletonRunnable.java:41)
at com.dremio.concurrent.SafeRunnable.run(SafeRunnable.java:40)
at com.dremio.concurrent.Runnables$1.run(Runnables.java:45)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
… 18 more
)
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-18.0.jar:na]
at com.dremio.exec.store.hive.HiveStoragePlugin.start(HiveStoragePlugin.java:575) ~[dremio-hive-plugin-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.ManagedStoragePlugin$2.run(ManagedStoragePlugin.java:238) ~[dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.concurrent.RenamingRunnable.run(RenamingRunnable.java:36) ~[dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.concurrent.SingletonRunnable.run(SingletonRunnable.java:41) ~[dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.concurrent.SafeRunnable.run(SafeRunnable.java:40) ~[dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.concurrent.Runnables$1.run(Runnables.java:45) ~[dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:420)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:236)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:181)
at com.dremio.exec.store.hive.HiveClient$1.run(HiveClient.java:144)
at com.dremio.exec.store.hive.HiveClient$1.run(HiveClient.java:141)
at com.dremio.exec.store.hive.HiveClient$9.run(HiveClient.java:316)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at com.dremio.exec.store.hive.HiveClient.doAsCommand(HiveClient.java:313)
at com.dremio.exec.store.hive.HiveClient.connect(HiveClient.java:140)
at com.dremio.exec.store.hive.HiveClient.createClient(HiveClient.java:126)
at com.dremio.exec.store.hive.HiveStoragePlugin.start(HiveStoragePlugin.java:573)
at com.dremio.exec.catalog.ManagedStoragePlugin$2.run(ManagedStoragePlugin.java:238)
at com.dremio.concurrent.RenamingRunnable.run(RenamingRunnable.java:36)
at com.dremio.concurrent.SingletonRunnable.run(SingletonRunnable.java:41)
at com.dremio.concurrent.SafeRunnable.run(SafeRunnable.java:40)
at com.dremio.concurrent.Runnables$1.run(Runnables.java:45)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
… 18 more
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:466) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:181) ~[hive-metastore-1.2.1.jar:1.2.1]
    at com.dremio.exec.store.hive.HiveClient$1.run(HiveClient.java:144) ~[dremio-hive-plugin-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
    at com.dremio.exec.store.hive.HiveClient$1.run(HiveClient.java:141) ~[dremio-hive-plugin-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
    at com.dremio.exec.store.hive.HiveClient$9.run(HiveClient.java:316) ~[dremio-hive-plugin-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
    at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_171]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_171]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807) ~[hadoop-common-2.8.0.jar:na]
    at com.dremio.exec.store.hive.HiveClient.doAsCommand(HiveClient.java:313) ~[dremio-hive-plugin-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
    at com.dremio.exec.store.hive.HiveClient.connect(HiveClient.java:140) ~[dremio-hive-plugin-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
    at com.dremio.exec.store.hive.HiveClient.createClient(HiveClient.java:126) ~[dremio-hive-plugin-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
    at com.dremio.exec.store.hive.HiveStoragePlugin.start(HiveStoragePlugin.java:573) ~[dremio-hive-plugin-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
    ... 5 common frames omitted
2018-06-13 07:29:37,378 [qtp283748905-110] INFO c.d.exec.catalog.CatalogServiceImpl - User Error Occurred [ErrorId: dab12d46-97b0-4ad2-a771-87bbf01c399f]
com.dremio.common.exceptions.UserException: Failure while configuring source [HDFS]. Info: Unavailable: Call From ip-172-30-0-244/127.0.0.1 to some_IP:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: ConnectionRefused - HADOOP2 - Apache Software Foundation
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.CatalogServiceImpl.createOrUpdateSource(CatalogServiceImpl.java:624) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:376) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.CatalogServiceImpl.access$600(CatalogServiceImpl.java:100) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.CatalogServiceImpl$SourceModifier.createSource(CatalogServiceImpl.java:946) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.CatalogImpl.createSource(CatalogImpl.java:529) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.DelegatingCatalog.createSource(DelegatingCatalog.java:182) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:147) [dremio-dac-backend-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:138) [dremio-dac-backend-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:134) [dremio-dac-backend-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:80) [dremio-dac-backend-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]

My java processes for hadoop are running, i.e. ResourceManager, NameNode, SecondaryNameNode, NodeManager, DataNode).

Hope somebody can help me out!

Thanks in advance!

anthony · June 13, 2018, 11:47am

Connection refused = networking/firewall issue. I would reinvestigate your AWS security policies. Per your post, telnet is a good way to see if the 2 servers can communicate through a specific port. If telnet fails, Dremio will likely fail.
I’m not sure if you are using AWS IP or hostname for the connection but may try the other?

isSokOl · June 15, 2018, 1:05pm

Hello @anthony,

thanks for your quick reply.

I was able to add HIVE. I had to start the hive metastore (obviously) and needed the process RunJar. Now connected to HIVE (however databases are still missing). Also, another weird thing: After starting the hive metastore service, I am able to reach the port via telnet. However, I still get an error “while fetching the resource: TypeError: NetworkError when attempting to fetch resource” using RESTED get request.

Also, still not able to add HDFS. Do I need another process for that (besides RecourceManager, NodeManager, NameNode, DataNode, SecondaryNameNode)?

isSokOl

anthony · June 15, 2018, 3:24pm

Can you send the new logs/stacktrace for the new Hive & HDFS connection issues? You don’t need any additional process than what you currently have for the HDFS connection.

isSokOl · June 18, 2018, 7:16am

Hello @anthony,

here are the new log files for the HDFS connection problem:

access.log

87.158.176.148 - - [18/Jun/2018:06:43:59 +0000] “PUT /apiv2/source/HDFS?nocache=1529304241178 HTTP/1.1” 400 314 “http://some_IP:9047/sources/list” “Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0”

server.log

2018-06-18 06:43:59,100 [qtp225293886-112] INFO c.d.exec.catalog.CatalogServiceImpl - User Error Occurred [ErrorId: a5aee6f7-cb41-4ddf-a4d2-1ea1d1593f16]
com.dremio.common.exceptions.UserException: Failure while configuring source [HDFS]. Info: Unavailable: Call From my_IP/127.0.0.1 to some_IP:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: ConnectionRefused - HADOOP2 - Apache Software Foundation
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.CatalogServiceImpl.createOrUpdateSource(CatalogServiceImpl.java:624) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:376) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.CatalogServiceImpl.access$600(CatalogServiceImpl.java:100) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.CatalogServiceImpl$SourceModifier.createSource(CatalogServiceImpl.java:946) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.CatalogImpl.createSource(CatalogImpl.java:529) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.catalog.DelegatingCatalog.createSource(DelegatingCatalog.java:182) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:147) [dremio-dac-backend-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:138) [dremio-dac-backend-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:134) [dremio-dac-backend-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:80) [dremio-dac-backend-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]

Also my results using telnet: Trying to telent ip 8020 gives me instant feedback that the connection could not be established. I had the exact same problem with hive until I started running the hive metastore.

The new HIVE issue, not seeing any databeses in Dremio, was resolved after logging in today.

Hope someone can help!

Best
isSokOl

anthony · June 18, 2018, 2:00pm

Can you share hdfs-site.xml? Can you also re-verify AWS firewall settings are open? The 2 nodes should be able to talk to each other via telnet

isSokOl · June 18, 2018, 3:21pm

The hdfs-site.xml file:

<configuration>
 <property>
  <name>dfs.replication</name>
  <value>1</value>
 </property>

 <property>
  <name>dfs.namenod.name.dir</name>
  <value>file:///usr/local/hadoop_store/namenode</value>
 </property>

 <property>
  <name>dfs.datanode.data.dir</name>
  <value>file:///usr/local/hadoop_store/datanode</value>
 </property>
</configuration>

In port in AWS is open.

anthony · June 18, 2018, 3:48pm

Can you also share core-site.xml? Can you verify that the Hadoop daemon is running & listening (netstat -tulpn)?

isSokOl · June 19, 2018, 7:36am

I have

$ jps
3092 SecondaryNameNode
3317 ResourceManager
4279 RunJar
2861 DataNode
2671 NameNode
6415 Jps
3471 NodeManager

and netstat -tulpn command gives

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:50090           0.0.0.0:*               LISTEN      3092/java
tcp        0      0 127.0.0.1:8020          0.0.0.0:*               LISTEN      2671/java
tcp        0      0 0.0.0.0:50070           0.0.0.0:*               LISTEN      2671/java
tcp        0      0 127.0.0.1:35479         0.0.0.0:*               LISTEN      2861/java
tcp        0      0 0.0.0.0:50010           0.0.0.0:*               LISTEN      2861/java
tcp        0      0 0.0.0.0:50075           0.0.0.0:*               LISTEN      2861/java
tcp        0      0 0.0.0.0:50020           0.0.0.0:*               LISTEN      2861/java
tcp6       0      0 :::8042                 :::*                    LISTEN      3471/java
tcp6       0      0 :::46701                :::*                    LISTEN      3471/java
tcp6       0      0 :::8088                 :::*                    LISTEN      3317/java
tcp6       0      0 :::13562                :::*                    LISTEN      3471/java
tcp6       0      0 :::8040                 :::*                    LISTEN      3471/java

Here is the core-site.xml file

<configuration>
 <property>
  <name>hadoop.tmp.dir</name>
  <value>/app/hadoop/tmp</value>
 </property>

 <property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:54310</value>
 </property>

  <property>
    <name>hadoop.proxyuser.dremio.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.dremio.groups</name>
    <value>*</value>
  </property>
</configuration>

Thanks for the help!

anthony · June 19, 2018, 1:09pm

According to your core-site.xml, you HDFS is running on port 54310, not the default 8020. Can you try that? Though I find it odd you have something listening on port 8020 (therefore telnet should still be successful).

isSokOl · June 19, 2018, 2:11pm

Changed port in core-site.xml to 8020. Now netstat -tulpn gives

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:50090           0.0.0.0:*               LISTEN      3243/java
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:35508         0.0.0.0:*               LISTEN      3017/java
tcp        0      0 127.0.0.1:8020          0.0.0.0:*               LISTEN      2824/java
tcp        0      0 0.0.0.0:50070           0.0.0.0:*               LISTEN      2824/java
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:50010           0.0.0.0:*               LISTEN      3017/java
tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:9083            0.0.0.0:*               LISTEN      4139/java
tcp        0      0 0.0.0.0:50075           0.0.0.0:*               LISTEN      3017/java
tcp        0      0 0.0.0.0:50020           0.0.0.0:*               LISTEN      3017/java
tcp6       0      0 :::8042                 :::*                    LISTEN      3748/java
tcp6       0      0 :::45678                :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -
tcp6       0      0 :::9047                 :::*                    LISTEN      -
tcp6       0      0 :::8088                 :::*                    LISTEN      3588/java
tcp6       0      0 :::13562                :::*                    LISTEN      3748/java
tcp6       0      0 ::1:6010                :::*                    LISTEN      -
tcp6       0      0 :::8030                 :::*                    LISTEN      3588/java
tcp6       0      0 :::8031                 :::*                    LISTEN      3588/java
tcp6       0      0 :::8032                 :::*                    LISTEN      3588/java
tcp6       0      0 :::8033                 :::*                    LISTEN      3588/java
tcp6       0      0 :::31010                :::*                    LISTEN      -
tcp6       0      0 :::2181                 :::*                    LISTEN      -
tcp6       0      0 :::39206                :::*                    LISTEN      3748/java
tcp6       0      0 :::8040                 :::*                    LISTEN      3748/java
udp        0      0 0.0.0.0:68              0.0.0.0:*                           -

With processes

6403 Jps
3588 ResourceManager
3748 NodeManager
2824 NameNode
3017 DataNode
4139 RunJar
3243 SecondaryNameNode

so Namenode is now listening on 8020.

Still can’t telnet the port. Still get the same error in dremio and server.log.

Topic		Replies	Views
Not able to connect HDFS to Dremio!	7	1794	December 16, 2020
Hive connection trouble	6	1427	September 3, 2019
Unable to connect to HDFS	7	1817	August 31, 2018
Dremio Connection With Remote Hive	14	2952	October 19, 2018
Not able to connect Hive Source	12	2110	March 20, 2021

Unable to add HDFS nor HIVE to DS

Related topics