Proposed new krb5 FILE ccache protocol

[Nico Williams]

On Tue, Jan 28, 2014 at 08:21:59PM -0500, Greg Hudson wrote:
> After discussing this with a few people, I don't think we would want an
> implementation of this in libkrb5, for a few reasons:

Thanks for your thoughtful reply.  Please bear with me below.

> * It carries a lot of complexity, some of which is user-visible.  What
> used to access only the named file would now create a directory
> alongside the file, which in edge cases can get left behind.

I think the only user-visible aspect is the edge cases where directories
get left behind, which shouldn't happen often and are of little concern
(they'd be in $TMPDIR, where one expects to find garbage).

> * It assumes that the process has write access to the parent directory
> of the ccache path.

The ancillary directory can be in $TMPDIR (we can assume at least /tmp),
and the main file can be written by truncation as a fallback (with all
the problems that that entails).

The ccache could be created in a directory just for that purpose (e.g.,
in the krb5_cc_new_unique() case, thus making it easier to identify and
automatically cleanup any garbage leaked in these edge cases.)

The ancillary directory could also be written as a fork, where that's
supported, thus making it even easier to clean up.

(It doesn't really matter where the ancillary directory is (especially
if the session keys stored in it are encrypted, and even more so if all
creds stored in it are encrypted), though it probably shouldn't be
written on, say, an NFS filesystem that requires Kerberos credentials to
access it, but that's true of ccache files in general, and especially
when the filesystem is accessed in the clear over the network.)

> * It makes assumptions about what kinds of creds are precious and what
> kinds are discardable.  Such assumptions can run afoul of unusual use
> cases, such as application servers using S4U2Self.

In the GSS cases S4U* tickets always go into new ccaches.

I see that kvno(1) has -U and -P options for S4U.  Is this actually used
in real life, or is this just for diagnostic purposes.  What
applications will search a ccache for S4U credentials specifically, as
opposed to using whatever credentials they find in the ccache?  If there
are any such applications, how do they keep their precious S4U tickets
alive?

> However, we would be happy to have a pluggable ccache layer.  With our
> plugin system, a pluggable ccache layer would naturally allow a
> dynamically loaded module to replace the built-in FILE type.

That's fair and very much appreciated.

> For the POSIX file locks problem, I think there are simpler solutions.
> For example, we could use fcntl plus flock on platforms where that
> works, and fcntl plus a global mutex (accepting the cost of serializing
> all ccache operations) on platforms where it doesn't.  In the long term
> the best answer is file-private locks[1], if that ever catches on.

That covers 99% of the cases, and all of the ones I care about today.
It doesn't have the other benefits of the new design, which I do want.

> For performance scalability, I think the best answer is a daemon-based
> ccache type, which is something we want anyway for better OS X
> integration.  I know that doesn't meet the requirement of interoperating
> with old code, but the complexity of maintaining FILE compatibility
> while also achieving scalable performance is just too high for us.  A
> ccache daemon has some relatively trivial options for achieving good
> performance, such as keeping file and memory copies of each ccache and
> using the memory ccache for reads.

Yes, this is highly desirable.  A decent non-MEMORY ccache is still
needed for the kcm case (daemon restarts should not wipe out
credentials, unless that's desired and the daemon configured to use the
MEMORY ccache type).

On IRC tonight we noticed at least one additional problem with the
current FILE implementation: it's supposed to be thread-safe to share a
single ccache with multiple threads, but it's not because of the
KRB5_TC_OPENCLOSE flag reset/restore business.  This can be fixed of
course.  Assuming a kcm is not threaded or that it has a ccache handle
per-thread then this wouldn't have to be fixed for the kcm case, but it
should still be fixed for the general case.

> Other changes I would like to have include:
> 
> * In the FILE cache implementation, marshal creds to memory and write
> them out all at once with O_APPEND.  That wouldn't give absolute
> guarantees, but in practice it should greatly reduce the incidence of
> interleaved or partially written credentials even in the face of failed
> locks and sudden process death.  This would also allow a substantial
> amount of code to be shared between cc_file.c and cc_keyring.c.

In the existing one, yes, this is desirable.

> * Better recovery for the FILE ccache type if we run into a partially
> written cred.  I'm not sure exactly what is required.

Pretend you've hit EOF.  Also truncate at the corrupted entry's start
offset, so as to clear up the corruption.

> * API changes for atomic ccache refresh (issue #7707), probably by
> improving krb5_cc_move().

Yes, write a MEMORY or new_unique ccache and krb5_cc_move() it into
place -- that's the only reasonable way to get atomicity out of this API
for writing.  You'll have to move part of the move operation into the
ccache plugin interface.

If you have rename(2) into place in mind... you'll need write permission
to the directory holding the ccache (see above).  Otherwise you must
continue to rely on file locking to make these things atomic.

> * API changes to supplant the TC_OPENCLOSE flag, which doesn't play well
> with threads.  At a minimum, a fix for issue #7804 (unsetting
> TC_OPENCLOSE causes writes to fail).

A transactional API for storing credentials would useful.  But
krb5_cc_move() and krb5_cc_copy_creds() are probably sufficient (if they
become transactional under the hood).

As for KRB5_TC_OPENCLOSE... I'd just make it a no-op and deprecate it.
Instead each cursor should get its own file descriptor for the ccache
for thread-safety, and the file should be re-opened at the start of any
search for creds -or when storing creds- if the file has changed
(st_dev, st_ino).  After all, that's actually the effect of
KRB5_TC_OPENCLOSE, and therefore its apparent purpose: notice ccache
re-initialization *except* when in the middle of iterating all creds
(since otherwise the offset of each cred becomes invalid if the file
changes, leading to apparent corruption).  (Also, re-opening the ccache
in the current design means dropping the locks that allow those offsets
to remain valid.)

The whole KRB5_TC_OPENCLOSE business is an implementation detail that
didn't need to be exposed.

Nico
--