Proposed new krb5 FILE ccache protocol

[Greg Hudson]

On 01/27/2014 05:50 PM, Nico Williams wrote:
> Below is a description of a new FILE ccache that is backwards
> interoperable and compatible with the current one, improves read
> performance and probably write performance as well, recovers from
> corruption, and is concurrency-safe without using POSIX file locking.

After discussing this with a few people, I don't think we would want an
implementation of this in libkrb5, for a few reasons:

* It carries a lot of complexity, some of which is user-visible.  What
used to access only the named file would now create a directory
alongside the file, which in edge cases can get left behind.

* It assumes that the process has write access to the parent directory
of the ccache path.

* It makes assumptions about what kinds of creds are precious and what
kinds are discardable.  Such assumptions can run afoul of unusual use
cases, such as application servers using S4U2Self.

However, we would be happy to have a pluggable ccache layer.  With our
plugin system, a pluggable ccache layer would naturally allow a
dynamically loaded module to replace the built-in FILE type.

For the POSIX file locks problem, I think there are simpler solutions.
For example, we could use fcntl plus flock on platforms where that
works, and fcntl plus a global mutex (accepting the cost of serializing
all ccache operations) on platforms where it doesn't.  In the long term
the best answer is file-private locks[1], if that ever catches on.

For performance scalability, I think the best answer is a daemon-based
ccache type, which is something we want anyway for better OS X
integration.  I know that doesn't meet the requirement of interoperating
with old code, but the complexity of maintaining FILE compatibility
while also achieving scalable performance is just too high for us.  A
ccache daemon has some relatively trivial options for achieving good
performance, such as keeping file and memory copies of each ccache and
using the memory ccache for reads.

Other changes I would like to have include:

* In the FILE cache implementation, marshal creds to memory and write
them out all at once with O_APPEND.  That wouldn't give absolute
guarantees, but in practice it should greatly reduce the incidence of
interleaved or partially written credentials even in the face of failed
locks and sudden process death.  This would also allow a substantial
amount of code to be shared between cc_file.c and cc_keyring.c.

* Better recovery for the FILE ccache type if we run into a partially
written cred.  I'm not sure exactly what is required.

* API changes for atomic ccache refresh (issue #7707), probably by
improving krb5_cc_move().

* API changes to supplant the TC_OPENCLOSE flag, which doesn't play well
with threads.  At a minimum, a fix for issue #7804 (unsetting
TC_OPENCLOSE causes writes to fail).

[1]
http://jtlayton.wordpress.com/2014/01/07/file-private-posix-locks-aka-un-posix-locks/