The joys of ActiveDirectory and Kerberized SSH

[Foley, Joe]

Hi there.

We had a hell of a time getting ActiveDirectory Kerberos and Kerberized
ssh to work here at iRobot, and I can't completely blame MicroSoft for
the problems.

Hacksaw (David Todd) best summarized it so I'm sending his email with
with a bit of rewording.

Goal: kerberize ssh access to our svn server (hq-svn), for use with
svn+ssh style URL's.

Involved software: libkrb53 in Ubuntu-8.04, ktpass from Microsoft on
MS-Server 2003, samba from Ubuntu 8.04.

Obstacles: There needs to be service principals put into the KDC, in
this case Microsoft's active directory.
   - The mechanisms for doing this don't tell you what you have done.
   - It's possible with the commands for adding principals to add them
to accounts to which they are not related. Thus (our Network Admin)
Keith, thinking he was adding SPNs to the hq-svn account was actually
adding them to his own accounts.

This is an important point. Keith was allowed to add things to the
database, which ensured that the database could not give a sane answer.
This is a defiency in AD, but it wouldn't surprise me if others made
this mistake.  

At some point the problem appeared to be the AD server giving back more
than one answer to a query on a principal.  This was very hard to detect
because different clients failed in very different ways.  Doing a kinit
with the keytab for the principal "hq-svn/host" would fail, but
"hq-svn%" worked.
	
Once we had figured that out, the next step was to try again. Once again
there was an error. In this case the error was "No such file or
directory."  with not hint as to what file or directory the thing might
have been looking for.

In fact, using ltrace didn't help this. I have no idea where it actually
looks up that file name.

It turns out that it was looking for keytab file. The only reason I
realized that was because I started trying to think of the mechanism,
and how it all worked.

We really could have used help from the kerberos clients error messages.
"No such file or directory" without context gives you no idea what to
fix.

When the clients got confused because they were getting multiple
responses from the AD server, they gave nonsensical error messages.  If
the clients had simply told us that they didn't like the multiple
answers it was getting back from the AD server, this would have helped
clear things up much faster.

To summarize, the error messages for problems like this *must* say what
exactly is wrong, what resource is wrong or missing, preferably with as
much context as possible.   It's not reasonable to hope that ltrace or
strace will give you enough information.  It is also not reasonable for
the sys-admin to be expected to run through the entire installation
procedure again.  Sys-admin's are harried, and need shortcuts and
complete error messages.


Joe Foley, Ph.D.
Senior Mechanical Engineer
iRobot G&I Research
Mail Stop: 8-1
8 Crosby Dr.
Bedford, MA 01730
T: 781-430-3117 
F: 781-430-3001