Query via Hive2 with HDFS not on 8020 issue

Dremio University
1

Hi,

I’ve found several other topics related to this issue. As adviced in https://community.dremio.com/t/issue-while-accessing-hive I’ve added core-site.xml and hive-site.xml to:

  • <DREMIO_HOME>/conf

    dremio@dremio-1:/opt/dremio/conf$ ls -lah
    total 44K
    drwxr-xr-x 1 root root 4.0K Sep 15 08:36 .
    drwxr-xr-x 1 root root 4.0K Aug 27 08:05 …
    -rwxrwxrwx 1 root root 2.1K Sep 15 08:35 core-site.xml
    -rw-r–r-- 1 root root 2.5K Aug 27 07:30 dremio-env
    -rw-r–r-- 1 root root 914 Aug 27 07:30 dremio.conf
    -rwxrwxrwx 1 root root 3.1K Sep 15 08:35 hive-site.xml
    -rw-r–r-- 1 root root 1.8K Aug 27 07:30 logback-access.xml
    -rw-r–r-- 1 root root 2.2K Aug 27 07:30 logback-admin.xml
    -rw-r–r-- 1 root root 7.5K Aug 27 07:30 logback.xml

  • <DREMIO_HOME>/plugins/connectors/hive2.d

    dremio@dremio-1:/opt/dremio/plugins/connectors/hive2.d$ ls -lah
    total 16K
    drwxr-xr-x 2 root root 4.0K Sep 15 08:36 .
    drwxr-xr-x 1 root root 4.0K Sep 15 08:36 …
    -rwxrwxrwx 1 root root 2.1K Sep 15 08:35 core-site.xml
    -rwxrwxrwx 1 root root 3.1K Sep 15 08:35 hive-site.xml

In my specific situation HDFS is listening on port 9000.

core-site.xml and hive-site.xml contains:

<property><name>fs.defaultFS</name><value>hdfs://namenode:9000</value></property>

Additionally i set

hive_connection
hive_connection1580×894 61.9 KB

Dremio lists the databases and tables, but as soon as I try to query I get the following message:

Call From dremio-1/172.23.0.11 to namenode:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

As you can see it still uses port 8020. As an attachment I’ve included the job profile.aa111dbc-f559-4683-9837-c0c7b8c9fc1e.zip (12.4 KB)

Any idea or hint where the namenode:8020, especially 8020, is comming from?

As an addition:

It seems to work for data persisted with Avro. But still throws this issue for Parquet data.

2

@gnr

Can you please send us the general page screen shot?

Thanks
Bali

3

Hi @balaji.ramaswamy ,

here the screenshot from the general page.

dremio_hive2_general
dremio_hive2_general1261×818 42.7 KB

4

@gnr

Instead of copying the xml files, can we just symlink the folder?

rm -rf <DREMIO_HOME>/plugins/connectors/hive2.d
ln -s <DREMIO_HOME>/conf <DREMIO_HOME>/plugins/connectors/hive2.d/conf

5

Hi, i’m getting the same error. Any luck figuring it out?

6

@borasy I have replied to you on Connect Dremio with hive-metastore running on localhost - #2 by thatran1

7

I’m able to reproduce this. I suspect this is a bug in the dremio hive plugins. Skimming the src, looks like Dremio just uses hadoop code/libraries to make the call to the namenode…but dremio needs to pass in the hdfs path, and i suspect that dremio does not pass in a path like

hdfs://<NAMENODE_HOST>:<PORT>/<HDFS_FILE_OR_DIR_PATH>

but rather just something like

<HDFS_FILE_OR_DIR_PATH>

so, the hadoop code assumes the default port. I can’t check it right now but will do so later today or tomorrow and update this thread with my findings.

1 Like