Can not access HDFS with keytab file （Resend）

Sun Jan 14 07:37:10 EST 2018

Hello,  I am  using Hadoop2.8 with Kerberos5 in the Docker, However I can
not access HDFS thought Kerberos ..The Docker image is
https://hub.docker.com/r/nlesc/xenon-hdfs-kerberos/
In the Docker container I have checked the `hdfs.keytab` by

> ```bash
> $ kinit -V  -kt /opt/hadoop/hdfs.keytab hdfs/localhost at esciencecenter.nl
> Using default cache: /tmp/krb5cc_0
> Using principal: hdfs/localhost at esciencecenter.nl
> Using keytab: /opt/hadoop/hdfs.keytab
> Authenticated to Kerberos v5
> ```

and here is the DEBUG info for command `hdfs dfs -ls /`

```
> hdfs dfs -ls /
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.hadoop.security.authentication.util.KerberosUtil
> (file:/opt/hadoop/share/hadoop/common/lib/hadoop-auth-2.8.1.jar) to method
> sun.security.krb5.Config.getInstance()
> WARNING: Please consider reporting this to the maintainers of
> org.apache.hadoop.security.authentication.util.KerberosUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> Java config name: null
> Native config name: /etc/krb5.conf
> Loading krb5 profile at /etc/krb5.conf
> Loaded from native config
> >>>KinitOptions cache name is /tmp/krb5cc_0
> >>>DEBUG <CCacheInputStream>  client principal is hdfs/
> localhost at esciencecenter.nl
> >>>DEBUG <CCacheInputStream> server principal is krbtgt/
> esciencecenter.nl at esciencecenter.nl
> >>>DEBUG <CCacheInputStream> key type: 17
> >>>DEBUG <CCacheInputStream> auth time: Sat Jan 13 03:31:02 GMT 2018
> >>>DEBUG <CCacheInputStream> start time: Sat Jan 13 03:31:02 GMT 2018
> >>>DEBUG <CCacheInputStream> end time: Sun Jan 14 03:31:02 GMT 2018
> >>>DEBUG <CCacheInputStream> renew_till time: null
> >>> CCacheInputStream: readFlags()  FORWARDABLE; INITIAL;
> >>>DEBUG <CCacheInputStream>  client principal is hdfs/
> localhost at esciencecenter.nl
> >>>DEBUG <CCacheInputStream> server principal is
> X-CACHECONF:/krb5_ccache_conf_data/fast_avail/krbtgt/esciencecenter.nl at esciencecenter.nl
> @esciencecenter.nl
> >>>DEBUG <CCacheInputStream> key type: 0
> >>>DEBUG <CCacheInputStream> auth time: Thu Jan 01 00:00:00 GMT 1970
> >>>DEBUG <CCacheInputStream> start time: null
> >>>DEBUG <CCacheInputStream> end time: Thu Jan 01 00:00:00 GMT 1970
> >>>DEBUG <CCacheInputStream> renew_till time: null
> >>> CCacheInputStream: readFlags()
> Found ticket for hdfs/localhost at esciencecenter.nl to go to krbtgt/
> esciencecenter.nl at esciencecenter.nl expiring on Sun Jan 14 03:31:02 GMT
> 2018
> Entered Krb5Context.initSecContext with state=STATE_NEW
> Found ticket for hdfs/localhost at esciencecenter.nl to go to krbtgt/
> esciencecenter.nl at esciencecenter.nl expiring on Sun Jan 14 03:31:02 GMT
> 2018
> Service ticket not found in the subject
> >>> Credentials acquireServiceCreds: same realm
> default etypes for default_tgs_enctypes: 17.
> >>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
> >>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
> >>> KdcAccessibility: reset
> >>> KrbKdcReq send: kdc=localhost UDP:88, timeout=30000, number of retries
> =3, #bytes=663
> >>> KDCCommunication: kdc=localhost UDP:88, timeout=30000,Attempt =1,
> #bytes=663
> >>> KrbKdcReq send: #bytes read=651
> >>> KdcAccessibility: remove localhost
> >>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
> >>> KrbApReq: APOptions are 00100000 00000000 00000000 00000000
> >>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
> Krb5Context setting mySeqNumber to: 481871376
> ...........
> drwxr-xr-x   - xenon supergroup          0 2018-01-13 02:15
> /filesystem-test-fixture
> drwxr-xr-x   - hdfs  supergroup          0 2018-01-13 04:39 /tmp
> ```

I use Scala to list the HDFS file with following code:

> ```scala
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> val conf = new Configuration()
> conf.set("fs.defaultFS", "hdfs://127.0.0.1:8020")
> conf.set("hadoop.security.authentication", "kerberos")
> UserGroupInformation.setConfiguration(conf)
>    UserGroupInformation.loginUserFromKeytab("hdfs/
> localhost at esciencecenter.nl","/opt/hadoop/hdfs.keytab")
> val fs = FileSystem.get(conf)
> val fsStatus = fs.listStatus(new Path("/"))
> ```

But it get wrong when run in the Docker container the error message is :

```bash
> WARNING: Illegal reflective access by
> org.apache.hadoop.security.authentication.util.KerberosUtil
> (file:/root/sbt-HttpExtract/lib_managed/jars/org.apache.hadoop/hadoop-auth/hadoop-auth-2.7.0.jar)
> to method sun.$
> ecurity.krb5.Config.getInstance()
> WARNING: Please consider reporting this to the maintainers of
> org.apache.hadoop.security.authentication.util.KerberosUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> [error] (run-main-0) java.io.IOException: Login failure for hdfs/
> localhost at esciencecenter.nl from keytab /opt/hadoop/hdfs.keytab:
> javax.security.auth.login.LoginException: Message stream modified (41)
> java.io.IOException: Login failure for hdfs/localhost at esciencecenter.nl
> from keytab /opt/hadoop/hdfs.keytab:
> javax.security.auth.login.LoginException: Message stream modified (41)
> at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:962)
>         at hdfs.sample.AuthDemo$.main(test.scala:23)
>         at hdfs.sample.AuthDemo.main(test.scala)
>         at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>         at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> Caused by: javax.security.auth.login.LoginException: Message stream
> modified (41)
>         at
> jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:781)
>         at
> jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:592)
>         at
> java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:726)
>         at
> java.base/javax.security.auth.login.LoginContext.access$000(LoginContext.java:194)
>         at
> java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665)
>         at
> java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663)
>         at java.base/java.security.AccessController.doPrivileged(Native
> Method)
>         at
> java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663)
>         at
> java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574)
>         at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:953)
>         at hdfs.sample.AuthDemo$.main(test.scala:23)
>         at hdfs.sample.AuthDemo.main(test.scala)
>         at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>         at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> Caused by: KrbException: Message stream modified (41)
> ```

I am new for Kerberos and I have worked for this problem for one day..I
really have no idea....
Please help me thx...