Problem integrating Dremio 4.1.4 with HDP 2.6

Hi Team,
we are evaluating the new Dremio 4.1.4 CE: we currently use Dremio 2.0.5 CE in our environments with no problem at all and we wish to upgrade our installation.

We are using official docker images with a simple deploy of one coordinator and one executor on two different containers. When configuring sources, we have trouble connecting with Hive Metastore (Hive 2.1 in HDP 2.6):

2020-02-11 14:54:22,772 [qtp302148962-95] DEBUG c.d.exec.store.hive.HiveConfFactory - Setting fs.s3.impl to org.apache.hadoop.fs.s3a.S3AFileSystem
2020-02-11 14:54:22,774 [start-hive] DEBUG c.d.e.catalog.ManagedStoragePlugin - Starting: hive
2020-02-11 14:54:22,811 [start-hive] INFO c.d.e.store.hive.HiveStoragePlugin - Setup Hadoop user info using kerberos principal dremio@sdp.csi.it and keytab file /opt/dremio/external/dremio.headless.keytab successful.
2020-02-11 14:54:22,812 [start-hive] INFO c.d.e.store.hive.HiveStoragePlugin - Hive Metastore SASL enabled. Kerberos principal: hive/sdnet-master3.sdp.csi.it@sdp.csi.it
2020-02-11 14:54:23,071 [qtp302148962-95] DEBUG c.d.e.store.hive.HiveStoragePlugin - Caught exception while trying to get status of HIVE source, error:
java.lang.NullPointerException: null
** at com.dremio.exec.store.hive.HiveStoragePlugin.getState(HiveStoragePlugin.java:664) ~[na:na]**

Could you give us a suggestion?
Till Dremio 3.x we hadn’t any problem configuring hive source with default configurations.

Thanks
Sergio

Hi @seraus

what is the value you provided for the “Hive Kerberos Principal” in the hive source. Can you provide complete stack or server.log file
@Venugopal_Menda

Hi Venugopal_Menda,
thanks for your answer.

We use hive/sdnet-master3.sdp.csi.it@sdp.csi.it as Hive Kerberos Principal in our definiton of dremio source

Here sdnet-master3.sdp.csi.it is the server (different form Dremio Server) where HiveServer2 and Hive Metastore are running.

Please find attached our server.log (coordinator)

Sergiocoordinator.zip (55.7 KB)

Hi @seraus

Can you just use “hive/_HOST@sdp.csi.it” and let me know.

@Venugopal_Menda

Same problem…
Please find the new log attached
thanks
coordinator-short.zip (11.2 KB)

I can give you some more information: today we checked that Dremio till version 4.0.4 is working with our deploy of HDP 2.6. So the problem occurs from version 4.0.5

Are there any relevant differences between Dremio 4.0.4 and Dremio 4.0.5 concerning Hive2/hive Metastore plugin?

Regards
Sergio

@seraus

There should not be any change. Is there a possibility that anything might have changed on the Hive side?