Connect Dremio to Hive backed by ADL in HDI

Tutorials
1

In order to connect Dremio successfully to a Hive table with data stored in ADL on HDI, there are a couple additional steps

  1. Copy adls2-oauth2-token-provider.jar (likely in /usr/lib/hdinsight-datalake/) to /opt/dremio/jars/3rdparty
  2. If Dremio node not is not part of the cluster (e.g. external VM) - Copy entire contents of /usr/lib/hdinsight-common from a cluster node to Dremio node
    If Dremio node is already part of the HDI cluster - Ignore this step
  3. Copy over core-site.xml (likely in /usr/hdp/current/hadoop-client/conf/) to /opt/dremio/conf
  4. Edit core-site.xml. Make sure the URLs from setting below are connectable (likely need to replace default host with IP)

<property> <name>fs.azure.datalake.token.provider.service.urls</name> <value>http://1.1.1.1:909/api/oauthtoken,http://1.1.1.1:909/api/oauthtoken</value> </property>

  1. Restart Dremio service
1 Like