klist versus rpc.gssd

[Matt Garman]

We're using NFSv4 with the sec=krb5p option for secure user home
directories.  This is on CentOS (RHEL) Linux, mixed 6.5 and 5.7
releases.  Kerberos flavor is MIT.

The other day, a user was getting "Permission denied" errors on the
entirety of the secure NFS mount.

Running "klist" showed he had a valid ticket well into the future.
(Sorry, I didn't think to capture the output of it.)

However, in the system log, I see entries like this:

Apr  1 06:00:12 client_server rpc.gssd[3465]: ERROR: GSS-API: error in
gss_acquire_cred(): The referenced credential has expired - No error
Apr  1 06:00:12 client_server rpc.gssd[3465]: WARNING: Failed while
limiting krb5 encryption types for user with uid 723
Apr  1 06:00:12 client_server rpc.gssd[3465]: WARNING: Failed to
create krb5 context for user with uid 723 for server nfs_server_name

The user's UID is 723.

This particular UID is actually a shared account used by a couple
people.  I noticed there were several /tmp/krb5cc_723_random files on
the machine.

It is possible that gss (or perhaps the Linux kernel) was referencing
much older credentials that truly did expire, despite what klist
reported?  That's the only plausible explanation I can think of... but
on the other hand, I have the ticket lifetime set to a period that is
much longer than the machine's uptime.  (In other words, the machine
had only been up for a week, and the ticket lifetime is much longer
than that, so how could an old/expired ticket show up?)

The simple workaround was do to a kdestroy, followed by a kinit.

I'd appreciate any thoughts anyone has, or other clues I might need to look for.

Thank you!
Matt