Problem integrating Dremio 4.1.4 with HDP 2.6

1

Hi Team,
we are evaluating the new Dremio 4.1.4 CE: we currently use Dremio 2.0.5 CE in our environments with no problem at all and we wish to upgrade our installation.

We are using official docker images with a simple deploy of one coordinator and one executor on two different containers. When configuring sources, we have trouble connecting with Hive Metastore (Hive 2.1 in HDP 2.6):

2020-02-11 14:54:22,772 [qtp302148962-95] DEBUG c.d.exec.store.hive.HiveConfFactory - Setting fs.s3.impl to org.apache.hadoop.fs.s3a.S3AFileSystem
2020-02-11 14:54:22,774 [start-hive] DEBUG c.d.e.catalog.ManagedStoragePlugin - Starting: hive
2020-02-11 14:54:22,811 [start-hive] INFO c.d.e.store.hive.HiveStoragePlugin - Setup Hadoop user info using kerberos principal dremio@sdp.csi.it and keytab file /opt/dremio/external/dremio.headless.keytab successful.
2020-02-11 14:54:22,812 [start-hive] INFO c.d.e.store.hive.HiveStoragePlugin - Hive Metastore SASL enabled. Kerberos principal: hive/sdnet-master3.sdp.csi.it@sdp.csi.it
2020-02-11 14:54:23,071 [qtp302148962-95] DEBUG c.d.e.store.hive.HiveStoragePlugin - Caught exception while trying to get status of HIVE source, error:
java.lang.NullPointerException: null
** at com.dremio.exec.store.hive.HiveStoragePlugin.getState(HiveStoragePlugin.java:664) ~[na:na]**

Could you give us a suggestion?
Till Dremio 3.x we hadn’t any problem configuring hive source with default configurations.

Thanks
Sergio

2

Hi @seraus

what is the value you provided for the “Hive Kerberos Principal” in the hive source. Can you provide complete stack or server.log file
@Venugopal_Menda

3

Hi Venugopal_Menda,
thanks for your answer.

We use hive/sdnet-master3.sdp.csi.it@sdp.csi.it as Hive Kerberos Principal in our definiton of dremio source

Here sdnet-master3.sdp.csi.it is the server (different form Dremio Server) where HiveServer2 and Hive Metastore are running.

Please find attached our server.log (coordinator)

Sergiocoordinator.zip (55.7 KB)

4

Hi @seraus

Can you just use “hive/_HOST@sdp.csi.it” and let me know.

@Venugopal_Menda

5

Same problem…
Please find the new log attached
thanks
coordinator-short.zip (11.2 KB)

6

I can give you some more information: today we checked that Dremio till version 4.0.4 is working with our deploy of HDP 2.6. So the problem occurs from version 4.0.5

Are there any relevant differences between Dremio 4.0.4 and Dremio 4.0.5 concerning Hive2/hive Metastore plugin?

Regards
Sergio

7

@seraus

There should not be any change. Is there a possibility that anything might have changed on the Hive side?

8

Is this issue fixed? we are also facing the same issue (unable to access the Hive source beyond the Dremio 4.0.4. The hive configured without SASL without any issues till the Dremio 4.0.4 version. The Dremio 4.0.5 onwards showing the following lines in server log

[main] INFO c.dremio.exec.catalog.PluginsManager - Result of storage plugin startup:
Hive: failed (775ms). Unavailable: null

9

Hi All,
i had the same issue trying to connect to an older HDP / Hive version, we have data in hdp 2.6 and from dremio 4.1.4 i could not connect so i went back to dremio version 4.0.4 and it worked.

would be good to know what has changed or if we just need to use an older version of dremio for connecting to legacy hdp/hive

10

@mstan

What version of Hive is this?

11

In hortonworks HDP 2.6 you had the option of hive version 1.x and 2.x, looks like we have Hive 1.2.1 in the cluster that i am trying to connect to. I guess Hive 1.x is not supported in 4.1.4