I am new to Dremio.
We have hive-metastore server running on localhost(thrift://localhost:9090), we want to establish connection between metastore server and Dremio.
Since hive metastore is following SSL, so we will need to provide following properties inside Dremio as part of creating connection:
Please let me know how can I supply mentioned properties in Dremio
Hi @lalitp ,
you supply the properties to dremio by
(1) copying your hive-site.xml, along with your hadoop’s core-site.xml, hdfs-site.xml, and mapred-site.xml into dremio’s conf directory and then also
(2) creating a
$DREMIO_INSTALL_DIR/plugins/connectors/<HIVE-PLUGIN-ID>/ directory, where
<HIVE-PLUGIN-ID> is one of:
depending on whether you’re using dremio OSS (no “-ee” in the hive plugin id) or dremio enerprise, and whether your datasource is hive 2 or hive 3 (see Dremio for more info).
After you create that directory, then
(3) Create this symlink:
$ ln -s $DREMIO_INSTALL_DIR/conf $DREMIO_INSTALL_DIR/plugins/connectors/<HIVE-PLUGIN-ID>/conf
For you/the dremio user to actually see the hive data in the Dremio UI, you need to have impersonation configured, as described here: Dremio
I follow everything here, but the dfs keeps defaulting to port 8020 although i set it to 9000 in core-site.xml
@borasy Is your HDFS Name Node port defaulting to 8020? Which property in core-site.xml did you change to 9000? What is the value of
The fs.defaultFS is set to hdfs://NameNodeHost:9000 in core-site.xml. So my Namenode listens to port 9000 i put this core-site.xml in dremio/conf. I linked this conf to hive3.d/conf. But Hive3 of dremio still wants to connect to 8020.
Do a diff of the following files:
Make sure they are all exactly the same.
Also, make sure the NameNode is actually listening on the port you think it’s listening on, using
$ sudo netstat -natp|grep <NameNode_PID>
It might also be worth checking your process owners’ environment variables:
$ env | grep -e 8020 -e 9000
I just now configured both dfs (hadoop 3.2.1) and dremio to use non-default port 9355 for the NameNode service and it worked just fine. So, please doublecheck your setup.
Check my update in Query via Hive2 with HDFS not on 8020 issue, i’m guessing that’s the problem you’re running into.