Connect Dremio with hive-metastore running on localhost

1

Hi Team,

I am new to Dremio.
We have hive-metastore server running on localhost(thrift://localhost:9090), we want to establish connection between metastore server and Dremio.
Since hive metastore is following SSL, so we will need to provide following properties inside Dremio as part of creating connection:

  • hive.metastore.use.SSL
  • hive.metastore.truststore.path
  • hive.metastore.truststore.password
  • hive.metastore.client.auth.mode
  • hive.metastore.client.plain.username
  • hive.metastore.client.plain.password

Please let me know how can I supply mentioned properties in Dremio

Thanks

2

Hi @lalitp ,
you supply the properties to dremio by

(1) copying your hive-site.xml, along with your hadoop’s core-site.xml, hdfs-site.xml, and mapred-site.xml into dremio’s conf directory and then also

(2) creating a $DREMIO_INSTALL_DIR/plugins/connectors/<HIVE-PLUGIN-ID>/ directory, where <HIVE-PLUGIN-ID> is one of:
hive2.d
hive3.d
hive2-ee.d
hive3-ee.d
depending on whether you’re using dremio OSS (no “-ee” in the hive plugin id) or dremio enerprise, and whether your datasource is hive 2 or hive 3 (see Dremio for more info).

After you create that directory, then
(3) Create this symlink:
$ ln -s $DREMIO_INSTALL_DIR/conf $DREMIO_INSTALL_DIR/plugins/connectors/<HIVE-PLUGIN-ID>/conf

For you/the dremio user to actually see the hive data in the Dremio UI, you need to have impersonation configured, as described here: Dremio

1 Like
3

I follow everything here, but the dfs keeps defaulting to port 8020 although i set it to 9000 in core-site.xml

4

@borasy Is your HDFS Name Node port defaulting to 8020? Which property in core-site.xml did you change to 9000? What is the value of fs.defaultFS?

5

The fs.defaultFS is set to hdfs://NameNodeHost:9000 in core-site.xml. So my Namenode listens to port 9000 i put this core-site.xml in dremio/conf. I linked this conf to hive3.d/conf. But Hive3 of dremio still wants to connect to 8020.

6

@borasy

Do a diff of the following files:

  • $HADOOP_INSTALL_DIR/etc/hadoop/core-site.xml
  • $DREMIO_INSTALL_DIR/conf/core-site.xml
  • $DREMIO_INSTALL_DIR/plugins/connectors/hive3.d/core-site.xml

Make sure they are all exactly the same.

Also, make sure the NameNode is actually listening on the port you think it’s listening on, using

$ sudo netstat -natp|grep <NameNode_PID>

It might also be worth checking your process owners’ environment variables:

$ env | grep -e 8020 -e 9000

I just now configured both dfs (hadoop 3.2.1) and dremio to use non-default port 9355 for the NameNode service and it worked just fine. So, please doublecheck your setup.

7

@borasy
Check my update in Query via Hive2 with HDFS not on 8020 issue, i’m guessing that’s the problem you’re running into.