Adding Multiple kerberos enabled HIVE, HDFS source in DREMIO

Hi Team,

Can we add the multiple kerberos enabled source in DREMIO. (like HDFS, HIVE).

When accessing any kerberos enabled source, we need to add the kerberos principal, keytab file path in dremio.conf file and core-site.xml, hdfs-site.xml, hive-site.xml files of that cluster inside the /etc/dremio/ directory.

With single source HDFS its working perfectly.

But when I tried with adding two different HDFS source in dremio by adding multiple keytab file path in dremio.conf file is not working, its only reading the one kaytab file.
Also I don’t how can I merger the core-site, hdfs-site.xml file of two hdfs sources.

Could you please help me how can I achieve this, what changes need to perform in core-site.xml, dremio.conf file.

Thanks & Regards,
Atul

Hi @achounde

First i would like to understand the setup for better visibility,

  1. Are you using YARN as executors ?. Or Dremio executors.

  2. How did you drefine the kerberos principal and ticket in Dremio.conf

  3. on the coordinator /etc/krb5.conf

Merging the files can make a conflict at multiple HDFS properties like default.fs . namenode information.

Hi Venugopal,

Thank you for quick reply. find the below information,

  1. I am using the DREMIO executors.

  2. How did you define the kerberos principal and ticket in Dremio.conf
    I have added the below property in dremio.conf file for access the single HDFS cluster.
    ===================
    services.kerberos: {
    principal: “testuser@DEV.COM”,
    keytab.file.path: “/etc/dremio/testuser.keytab”
    }
    ===================
    Also copied the core-site.xml, hdfs-site.xml inside /etc/dremio/ directory.
    For single node its working perfectly. but what should i do for multiple HDFS source with different key tab and principals.

  3. on the coordinator /etc/krb5.conf

  • Yes I have edited the krb5.conf file on coordinatior node and added the details of HDFS nodes.

Do you have any sample dremio configuration that used for accessing multiple HDFS cluster with kerberos.

Hi Team,

Do you have any idea how can I add multiple HIVE, hddfs source with kerberos enabled in same dremio cluster.

@achounde

You would have to add the properties individually and see if it works

Thanks
Bali

I have tried with adding the distinct properties while adding source on dremio portal.

my source cluster are kerberos enabled and dremio reading only one *.keytab file