KrbException: Identifier doesn't match expected value (906)

[lune voo]

 Hello everyone.

I am using a old version of IPA (3.0) on a RHEL6.6. (I know it is old ^_^).

I send you this mail because I had a problem during the night during the
execution of a spark job.
The error which occured was the following :

19/09/05 03:06:01 INFO Client: Application report for <MYAPPLICATION>
(state: FINISHED)
19/09/05 03:06:01 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: <APPMASTERIP>
         ApplicationMaster RPC port: 0
         queue: <QUEUE>
         start time: 1567040464637
         final status: SUCCEEDED
         tracking URL: http://<RM_HOST>:<RM_PORT>/proxy/<MYAPPLICATION2>/
         user: <myuser>
Exception in thread "main" java.io.IOException: Login failure for <myuser>
from keytab <LOCAL_PATH_FOR_KEYTAB>/<myuser>.headless.keytab:
javax.security.auth.login.LoginException: Generic error (description in
e-text) (60) - HANDLE_AUTHDATA
        at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1351)
        at
org.apache.spark.deploy.yarn.Client.cleanupStagingDir(Client.scala:210)
        at
org.apache.spark.deploy.yarn.Client.monitorApplication(Client.scala:1204)
        at org.apache.spark.deploy.yarn.Client.run(Client.scala:1258)
        at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1307)
        at org.apache.spark.deploy.yarn.Client.main(Client.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:751)
        at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
        at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: javax.security.auth.login.LoginException: Generic error
(description in e-text) (60) - HANDLE_AUTHDATA
        at
com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804)
        at
com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        at
javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
        at
javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
        at
javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at
javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1340)
        ... 14 more
Caused by: KrbException: Generic error (description in e-text) (60) -
HANDLE_AUTHDATA
        at sun.security.krb5.KrbAsRep.<init>(KrbAsRep.java:82)
        at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316)
        at
sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)
        at
com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)
        ... 27 more
Caused by: KrbException: Identifier doesn't match expected value (906)
        at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
        at sun.security.krb5.internal.ASRep.init(ASRep.java:64)
        at sun.security.krb5.internal.ASRep.<init>(ASRep.java:59)
        at sun.security.krb5.KrbAsRep.<init>(KrbAsRep.java:60)
        ... 30 more
19/09/05 03:06:01 INFO ShutdownHookManager: Shutdown hook called

The error occurred after the end of a spark job in fact.
I was wondering if this kind of error means something for you ?

I checked on google and saw the problem could come :

   1. from the krb5.conf file
   2. from the keytab
   3. from the fact that the login of <myuser> is mispelled during the TGT
   step
   4. from the encryption between the client and the server


As a note at the beginning of the job, there is a kinit performed on this
host and the kinit is OK.
So I checked the krb5.conf and it is OK.
I checked the keytab file and it is OK.

I was wondering if I could see maybe more information on the client side.
Do you know where are located the logs of a kinit on the client side please
?
Do you know which logs I could check on server side please ?

Best regards.

Lune