Hadoop Datanode service throws exception with Kerberos security enabled

[Sonia Garudi]

Hello team,
We have a Ambari cluster setup using Rhel 7.5 beta machines. We are facing
issues with start up of Hadoop Datanode on enabling Kerberos security.

Error logged in /var/log/krb5kdc.log -
Mar 27 14:48:17 pts00433-vm38.persistent.co.in krb5kdc[8737](info): TGS_REQ
(1 etypes {16}) 10.77.67.132: PROCESS_TGS: authtime 0,
dn/pts00433-vm38.persistent.co.in at EXAMPLE.COM for
nn/pts00433-vm38.persistent.co.in at EXAMPLE.COM, Ticket expired
Mar 27 14:48:55 pts00433-vm38.persistent.co.in krb5kdc[8737](info): TGS_REQ
(4 etypes {18 17 16 23}) 10.77.67.132: PROCESS_TGS: authtime 0,
nn/pts00433-vm38.persistent.co.in at EXAMPLE.COM for
nn/pts00433-vm38.persistent.co.in at EXAMPLE.COM, Ticket expired

Below error in service log:
2018-03-27 14:46:44,739 WARN  ipc.Client (Client.java:run(711)) - Couldn't
setup connection for dn/pts00433-vm38.persistent.co.in at EXAMPLE.COM to
pts00433-vm38.persistent.co.in/10.77.67.132:8020
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Ticket
expired (32) - PROCESS_TGS)]

We have following packages installed :
Version-Release number of selected component (if applicable):
# yum list installed | grep krb
krb5-devel.ppc64le                 1.15.1-18.el7       installed
krb5-libs.ppc64le                  1.15.1-18.el7       @anaconda/7.5
krb5-pkinit.ppc64le                1.15.1-18.el7       installed
krb5-server.ppc64le                1.15.1-18.el7       installed
krb5-workstation.ppc64le           1.15.1-18.el7       installed

# krb5-config --version
Kerberos 5 release 1.15.1

System and Ambari cluster details :
# uname -a
Linux pts00433-vm38.persistent.co.in 3.10.0-830.el7.ppc64le #1 SMP Mon Jan
15 12:26:57 EST 2018 ppc64le ppc64le ppc64le GNU/Linux
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 Beta (Maipo)

Ambari version : 2.6.1
HDP version installed : 2.6.4

We have noticed, with Kerberos build version 1.15.1-8.el7, the datanode
starts up without any issue.

Any help or suggestions on why it fails with the higher update would be
appreciated .

Regards,
Sonia Garudi
sgarudi at us.ibm.com