Unable to connect to hdfs with impersonation feature

1

Hi,

currentl, I’m trying to use dremio to connect hdfs ,I have tried below link to configure dremio or my HDP 3.1 , but with no luck.

https://docs.dremio.com/data-sources/hdfs.html

the server log showing below error org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Unauthorized connection for super-user: root from IP xx.xx.xx.xx

could anyone please tell me why dremio still use root user to connect hdfs instead of other user when I enable the impersonation feature? in my dremio cluster , I use root to start dremio

2

@vincent

The Dremio process user still needs access to HDFS to read data - Dremio will use impersonation to check if the current user has access to the data, but the actual reading is done as the process user.

1 Like
3

@doron thanks so much for the reply. really appreate the quick reaspond.

so the process owner is the user which I login dremio. also , I have to create the user in Linux with hdfs read privileage. is that correct?

4

Hi all

I did a workaround to fix the issue . I exchanged admin user to start dremio service which that mean I will use admin user to connect hive databases and also I will use the same user to access hdfs, all the user is going to access will be controled by my ranger. even I done this, I stil faced two below issue

  1. unable to reconise my hdp cluster
  2. unable to fetch chinese character

after, searching and troubleshotting, I copy hdfs-site,hive-site to my dremio conf folder which sovle the first issue. and I added one praremeter in dremio-env under conf (DREMIO_JAVA_SERVER_EXTRA_OPTS=’-Dsaffron.default.charset=UTF-16LE -Dsaffron.default.nationalcharset=UTF-16LE -Dsaffron.default.collation.name=UTF-16LE$en_US’) which solve the second issue.

from now on, the dremio can work properly connected to remote hdp cluster

5

In HDFS you have to do what the article you linked says: in HDFS, enable proxy user for the process user and then allow that user to proxy as the users who you need impersonation for (or do a * to allow all).