klist versus rpc.gssd

[Simo Sorce]

On Thu, 2015-04-02 at 10:45 -0500, Matt Garman wrote:
> We're using NFSv4 with the sec=krb5p option for secure user home
> directories.  This is on CentOS (RHEL) Linux, mixed 6.5 and 5.7
> releases.  Kerberos flavor is MIT.
> 
> The other day, a user was getting "Permission denied" errors on the
> entirety of the secure NFS mount.
> 
> Running "klist" showed he had a valid ticket well into the future.
> (Sorry, I didn't think to capture the output of it.)
> 
> However, in the system log, I see entries like this:
> 
> Apr  1 06:00:12 client_server rpc.gssd[3465]: ERROR: GSS-API: error in
> gss_acquire_cred(): The referenced credential has expired - No error
> Apr  1 06:00:12 client_server rpc.gssd[3465]: WARNING: Failed while
> limiting krb5 encryption types for user with uid 723
> Apr  1 06:00:12 client_server rpc.gssd[3465]: WARNING: Failed to
> create krb5 context for user with uid 723 for server nfs_server_name
> 
> The user's UID is 723.
> 
> This particular UID is actually a shared account used by a couple
> people.  I noticed there were several /tmp/krb5cc_723_random files on
> the machine.
> 
> It is possible that gss (or perhaps the Linux kernel) was referencing
> much older credentials that truly did expire, despite what klist
> reported?  That's the only plausible explanation I can think of... but
> on the other hand, I have the ticket lifetime set to a period that is
> much longer than the machine's uptime.  (In other words, the machine
> had only been up for a week, and the ticket lifetime is much longer
> than that, so how could an old/expired ticket show up?)
> 
> The simple workaround was do to a kdestroy, followed by a kinit.
> 
> I'd appreciate any thoughts anyone has, or other clues I might need to look for.

When rpc.gssd crawls /tmp for tickets, it will take the first one that
matches the uid, and try to use it. If it fails ... though luck.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York