ssh gssapi-with-mic and "Key table entry not found"

[Matt Garman]

On Tue, Aug 7, 2012 at 11:40 PM, Greg Hudson <ghudson at mit.edu> wrote:
> On 08/07/2012 01:23 PM, Matt Garman wrote:
>>     [root at lnxsvr11 ~]# grep 192.168.187.67 /etc/hosts
>>     192.168.187.67     lnxsvr11.mydomain.com      lnxsvr11
>>     [root at lnxsvr11 ~]# grep "lnxsvr11\." /etc/hosts
>>     192.168.187.67     lnxsvr11.mydomain.com      lnxsvr11
>>     [root at lnxsvr11 ~]# grep "lnxsvr11$" /etc/hosts
>>     192.168.187.67     lnxsvr11.mydomain.com      lnxsvr11
>
> I didn't see a smoking gun in the output you sent, but here are some
> pointers.  You need the following three things to match:
>
>   1. The canonicalization of lnxsvr11 on the client
>   2. The canonicalization of the result of gethostname() on the server
>   3. The hostname in the keytab entry
>
> By default, canonicalization is done using getaddrinfo() forward
> resolution and then getnameinfo() reverse resolution.  Depending on the
> Kerberos version, canonicalization may or may not be restricted to IPv4
> lookups.  (2) can be ignored if your sshd has sxw's patches using the
> "GSSAPIStrictAcceptorCheck no" option, but I don't think CentOS 5.7's
> build of sshd has those patches.
>
> You can see what (1) is by running klist on the client after attempting
> the ssh.  That would be useful information for debugging this problem.
>
> You should be able to see what (2) is on the server by getting tickets
> for somebody and then running "kvno -S host lnxsvr11" and seeing what
> principal that gets (or tries to get) server tickets for.  (There are
> easier ways to do this with more recent versions of MIT Kerberos--in 1.9
> or later, it would appear in the sshd debug output, for instance.  But
> CentOS only has krb5 1.6.1 as far as I can tell.)
>
> It's of course easy to see what (3) is, and you've already provided that.

Now it is magically working, and in fact, it is working for all machines.

I was having the problem yesterday, and the last thing I did was send
the last email to this list.  I went home, came in this morning to
work through your suggestions, but the problem no longer exists.

I tried another machine: it didn't work, but I didn't have the
"host/hostname" principal's key exported to the keytab file.  So I
went through all machines and made sure they had the "host/hostname"
key in the keytab.

On that last remaining non-working target machine, I did a "klist
-kte" and realized I had exported the "nfs/hostname" key to the keytab
twice, instead of "nfs/hostname" and "host/hostname".  I fixed the
keytab, restarted sshd, yet the problem persisted.  So I was gearing
up to work through your suggestions, but I thought maybe my original
TGT (on lnxsvr3) was somehow messed up.  So I did a kdestroy followed
by kinit, then logged into that machine successfully.

I don't know enough about how Kerberos works, but I'll speculate a
guess as to what was wrong yesterday: after a failed gssapi-with-mic
login attempt, some "residual stuff" gets attached to the original
TGT, some kind of "cache" of the "permission denied" situation.  IOW,
no matter what I do on the target machine, that the client still
"remembers" it didn't work means it continues to not work, until the
ticket is destroyed or expires (and my tickets are currently set to
the default 24h expiry).  Note that all that is a guess, maybe the
more knowledgeable folks can confirm (assuming you can make sense of
what I'm trying to convey!).

If nothing else, it seems that when troubleshooting these kinds of
issues in the future, I should always do a kdestroy, kinit between
each ssh attempt.

Thanks again for your help Simo and Greg.
-Matt