Executors are failing when deployed to kubernetes

Hey,

I’ve deployed Dremio with default config settings onto k8s. Dremio master works (I’m able to add sources), but I do have a problem with executors. They are crashing right after start with:

Dremio is exiting. Failure while starting services.
org.apache.hadoop.security.KerberosAuthException: failure to login: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name

I’m using original dremio-oss 18.1 docker image.
Can somebody help here?

Full stacktrace:

2021-10-14 11:27:06,221 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
org.apache.hadoop.security.KerberosAuthException: failure to login: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
	at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
	at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
	at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
	at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
	at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
	at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3487)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3477)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3319)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
	at com.dremio.exec.store.dfs.PDFSService.createFileSystem(PDFSService.java:99)
	at com.dremio.exec.store.dfs.PDFSService.start(PDFSService.java:93)
	at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
	at com.dremio.dac.daemon.NonMasterSingletonRegistry.start(NonMasterSingletonRegistry.java:54)
	at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:196)
	at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:202)
	at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:146)
	at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847)
	at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3487)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3477)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3319)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
	at com.dremio.exec.store.dfs.PDFSService.createFileSystem(PDFSService.java:99)
	at com.dremio.exec.store.dfs.PDFSService.start(PDFSService.java:93)
	at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
	at com.dremio.dac.daemon.NonMasterSingletonRegistry.start(NonMasterSingletonRegistry.java:54)
	at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:196)
	at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:202)
	at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:146)
Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name

@marek Maybe this SO thread will help? authentication - Hadoop IOException failure to login - Stack Overflow