krb5 libraries and circular dependencies

[ghudson@MIT.EDU]

I have two tenuously related library issues to discuss.

1. Currently we have libkrb5, libk5crypto (used by libkrb5), and
libkrb5support (used by both).  This separation creates some
chicken-and-egg issues for facilities which are useful to both libkrb5
and libk5crypto, such as error handling and trace logging.  For
example, consider trace logging's custom formatter:

  * If it lives in libkrb5, libk5crypto can't use it.
  * If it lives in libkrb5support, it can't use krb5 functions like
    krb5_unparse_name).
  * If it lives in libk5crypto, everything works, but that's inelegant
    if the facility has nothing to do with crypto.

So, I'd like to raise the idea of combining all three libraries into
one.  The question here is whether libk5crypto is separate from
libkrb5 purely due to erstwhile export regulations, or whether we want
libk5crypto (eventually) be useful independently of libkrb5, like the
OpenSSL crypto library is.

Currently libk5crypto is pretty tightly coupled to libkrb5.  Its
prototypes are in krb5.h, and its functions take libkrb5 contexts
(which can only be initialized from libkrb5).  On the other hand, it
does tolerates NULL context parameters, and I have seen interest in
being able to use it as an RFC 3961 implementation independently of
Kerberos.  I don't know if having to link in all of libkrb5 would be
a real impediment to that.

2. Plugins loaded from libkrb5 can make use of libkrb5--this kind of
circular dependency is allowed when you dynamically load things--and
they quite often do so.  However, there is a risk that the plugin will
be linked against a different copy of libkrb5 than the one which
loaded the plugin, with the following consequences:

  * If the plugin passes a pointer to a private data structure (like a
    krb5_context) to a different copy of libkrb5, it might be
    interpreted incorrectly, likely leading to a crash.
 
  * The two libkrb5s will have disjoint global data areas.  So, for
    instance, they will have different memory ccache lists.  The worst
    case where would be a global mutex protecting a POSIX file lock
    (on a replay cache, for instance)--that only works if the mutex is
    truly global.

There are three directions we can go in here:

  * We can try to make it unnecessary (and disallowed) for plugins to
    directly use libkrb5, by providing them with callbacks.  In the
    extreme case, we could provide plugins with a giant vtable
    containing all libkrb5 APIs.  Maintaining such a vtable would be a
    bit of a hassle.

  * We can allow plugins to use libkrb5, and create a safety mechanism
    to fail explicitly if a plugin was linked against a different copy
    of libkrb5 than the one which loaded it.  To do this, we'd ask
    plugins to call a libkrb5 function which compares the address of a
    global variable or function to one provided to the plugin.

  * We can allow plugins to use libkrb5, and try to make things
    actually work if the plugin was linked against a different copy of
    libkrb5.  I don't think this is possible in the general case; see
    the aforementioned example of the global mutex protecting a POSIX
    file lock.

I feel like currently we're awkwardly tugging in the first and second
directions (without the safety mechanism).  The preauth plugin
framework has the preauth_get_client_data_proc callback, for instance,
but it's not enough to keep pkinit from needing to make calls into
libkrb5 (both API calls and internal calls through the accessor).

Comments appreciated.