Dremio-admin restore doesn't work in kerberized environment

1

Hi,

dremio ver 4.2.2(all other prvs versions)
distributed storage: hdfs with kerberos authentication
dremio deployed in k8 using helm chart

backup command works and can create backup on hdfs dir or local dir
restore command fails with both local dir/hdfs dir

using local backupdir:

dremio@dremio-admin:/opt/dremio/bin$ ./dremio-admin restore -d /opt/dremio/dremio_backup/dremio_backup_2020-05-09_15.23/ -v
Restore failed
java.io.IOException: DestHost:destPort <NAME_NODE>:8020 , LocalHost:localPort dremio-admin/10.42.2.17:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

using hdfs backup dir:

dremio@dremio-admin:/opt/dremio/bin$ ./dremio-admin restore -d hdfs:/${NAME_NODE}:8020/user/dremio_backup/dremio_backup_2020-05-09_15.23/ -v
Restore failed
java.io.IOException: DestHost:destPort <NAME_NODE>:8020 , LocalHost:localPort dremio-admin/10.42.2.17:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

kerberos configuration:

dremio.conf
services: {
          kerberos.principal: "dremio@DOMAIN.COM"
          kerberos.keytab.file.path: "/opt/dremio/secret/.dremio.keytab"

}


dremio@dremio-admin:/opt/dremio$ ls -al secret/
total 4
drwxr-xr-x. 2 root   root    32 May  9 19:44 .
drwxr-xr-t. 1 root   root    20 May  9 19:44 ..
-r--------. 1 dremio dremio 413 May  9 19:44 .dremio.keytab
dremio@dremio-admin:/opt/dremio$ 

dremio@dremio-admin:/opt/dremio/conf$ ls -l
total 0
lrwxrwxrwx. 1 root root 20 May  9 19:44 core-site.xml -> ..data/core-site.xml
lrwxrwxrwx. 1 root root 17 May  9 19:44 dremio-env -> ..data/dremio-env
lrwxrwxrwx. 1 root root 18 May  9 19:44 dremio.conf -> ..data/dremio.conf
lrwxrwxrwx. 1 root root 20 May  9 19:44 hdfs-site.xml -> ..data/hdfs-site.xml
lrwxrwxrwx. 1 root root 16 May  9 19:44 krb5.conf -> ..data/krb5.conf
lrwxrwxrwx. 1 root root 25 May  9 19:44 logback-access.xml -> ..data/logback-access.xml
lrwxrwxrwx. 1 root root 24 May  9 19:44 logback-admin.xml -> ..data/logback-admin.xml
lrwxrwxrwx. 1 root root 18 May  9 19:44 logback.xml -> ..data/logback.xml
dremio@dremio-admin:/opt/dremio/conf$ 

dremio@dremio-admin:/opt/dremio/conf$ echo $DREMIO_JAVA_EXTRA_OPTS
-Djava.security.krb5.conf=/opt/dremio/conf/krb5.conf -Ddremio.flight.host=0.0.0.0 -Ddremio.flight.enabled=true -Ddremio.flight.parallel.enabled=true
dremio@dremio-admin:/opt/dremio/conf$
2

Hi @smora

Can you check whether you have a valid ticket, Can you do kinit and try, please share server.log on the next failure.
@Venugopal_Menda

3

Hi @Venugopal_Menda

I am running dremio in a k8s env, dremio container doesnt have kinit installed.
backup works from dremio-master pod, only restore is not able to initialize kerbereos.
dremio-admin pod is started using provided helm chart

for restore all dremio services are brought down and container is started in admin mode, I dont see any server.log. I do see admin.log but this is all empty

admin log:

dremio@dremio-admin:/$ cat var/log/dremio/admin_restore_2020-05-11_14*.log
dremio@dremio-admin:/$

backup from dremio-master

dremio@dremio-master-0:/opt/dremio/bin$ ./dremio-admin backup -a -l -d hdfs://${NAME_NODE}:8020/user/dremio_backup
dremio@dremio-master-0:/opt/dremio/bin$ 
dremio@dremio-master-0:/opt/dremio/bin$ kinit
bash: kinit: command not found