thread-safe Kerberos libraries

[Ken Raeburn]

Emily Ratliff <ratliff at austin.ibm.com> writes:
> Now that you've forgotten what you said:
> http://diswww.mit.edu:8008/menelaus.mit.edu/krb5dev/6761
> 
> and I've played with it for a while, I have a few questions and comments.

Oh, I haven't forgotten.  In fact, just a few days ago I was saying to
Tom that I should email you and find out how it was going....

> > The callback functions may only be registered while no locks are held
> > (which presumably means before any threads are created and while in
> > the main application).  Locks are not held across library calls; they
> > are always released.
> I agree with both of these items, but the first is the goal of the 
> application. I don't see how to enforce this in the library.

Yes, we would just assert that it's the application writer's
responsiblity.  If an application dynamically loads the library after
spawning multiple threads, we may have a problem.  I don't know if
there's anything we can do about that.

> > (Question: Should the callback functions be registered through
> > separate functions, or one call with extra arguments or a structure
> > pointer?)
> OpenSSL registers the callback functions individually, but I still prefer 
> a structure pointer.

Okay.  I don't think it makes a lot of difference.  Both approaches
are extensible in different ways, and we probably don't want much in
the way of extensions happening anyways.

> > Each library (each that has data needing protecting, that is) provides
> > a set of functions for manipulating these callbacks, based in part on
> > the OpenSSL API.
> Why allow the capability to have different callbacks for different 
> libraries loaded by the same application? In other words, if a single 
> application uses more than one library, the app has to register the same 
> callbacks with each libary that it uses. OpenSSL only has the thread 
> support in one libary, so this issue hasn't shown up for them.

Actually, I was thinking that every library would have to be using the
same thread interface through callbacks.  But if an application only
directly refers to one of the APIs we provide (GSS, krb5, RPC after we
get rpcsec_gss support), they shouldn't need to use the callback
registration function from a different API.  E.g., a "pure" GSS
application shouldn't call or even know about
krb5_register_thread_callback_thingamabob, it should only call
gss_register_thread_callback_thingamabob.

But at the same time, the GSS and krb5 thread hooks shouldn't have
compiled-in knowledge of, for example, how many locks are needed by
the other library or what thread package support was compiled in.  So
I'd expect the GSS library to act like any other krb5 application,
registering thread callback functions if it's told to use something
other than the default (or perhaps always registering?), reporting
that GSS is not thread-safe if krb5 isn't, etc.

This sort of layering requires a few tricks, for example gss_num_locks
would need to ask krb5_num_locks how many locks the krb5 library
needs, and add them to the count of locks that the GSS library itself
needs.


> > (Question: Should we consider gssapi thread-safe if it uses locks
> > around krb5 calls that can call DNS or C library routines that are not
> > known to be thread-safe, or worse, are known not to be thread-safe?)
> Sure, but hopefully we will do better.

I'm not so sure.  Okay, so we put a lock around GSS calls to
gethostbyname.  And we put another lock around krb5 calls to
gethostbyname.  If the problem isn't already obvious, even combining
those locks into one won't prevent another application thread from
calling gethostbyname.

I guess it comes down to, what does "the krb5 library is thread-safe"
mean?  Can it be used in one thread while another thread is doing
random C library calls?  Or must C library calls be excluded while the
krb5 library is being used (in however many threads)?

To put a different spin on it, if some other package maintainer uses
the same definition for "thread-safe" that we do, can their library
and ours be used simultaneously in different threads?  Or do two
supposedly thread-safe libraries add up to an unsafe application?

Note that I'm assuming we get stuck with some C library or resolver
functions not known to be thread-safe, or known not to be; if we get
rid of all such calls on a system, obviously we win big.  But I'm
fairly confident that on some systems we just won't be able to.

> > Should locks be ref-counted?  
> ...
> > (What's OpenSSL do?)
> OpenSSL does do ref-counting for locks. I'd like to postpone this one til 
> later though.

Okay.

> > For each library's prefix "foo" (to be determined for libraries like
> > com_err without consistent prefixes), we'd have the following
> > functions and macros, with names adjusted accordingly:
> 
> > void foo_set_locking_callback (void (*lockfn)(int mode, int locknum,
> >                                               const char *file,
> >                                               int line));
> Why per-library on this one? Why not a global that is shared? It will be 
> set the same unless the application is doing something very strange. 

What name should we use for this, that would be appropriate for a pure
krb5 application and for a pure gssapi application and for a pure
sunrpc application and for a pure krb4 application (hopefully a dying
breed, but still out there), and an application using com_err but none
of the rest of Kerberos (as happens on Linux, for example)?

This routine could be global, but only if the lock numbers for
everything are going to be coordinated.  Someday, I'd like to see us
have a GSS library capable of dynamically loading multiple mechanisms
from the filesystem.  It's hard to indicate the correct number of
locks if the implementations of different mechanisms don't coordinate.

I sort of anticipated doing something like:

    #define gss_locks_needed  8
    static void (*lockfn)() = ...;
    int gss_num_locks () {
        return gss_locks_needed + krb5_num_locks ();
    }
    static void k5lock (int mode, int locknum, const char *file, int line) {
        (*lockfn) (mode, locknum + gss_locks_needed, file, line);
    }
    void gss_set_locking_callback (void (*fn)()) {
        lockfn = fn;
        krb5_set_locking_callback (k5lock);
    }

This lets a GSS application register a callback and go about its
business.  And if a GSS/krb5 application wants to manage the krb5
locks directly, that'll work too, as long as the same thread package
underlies all the registered callbacks.  It would just mean the GSS
code reserves some lock numbers that don't get used.

This can be extended to deal with dynamically loaded objects, though
they'd probably all have to be loaded just to give an answer for
gss_num_locks if the info isn't cached in a config file or
something....


> >    Q: What about systems that might have multiple thread packages that
> >    *are* known to play nicely together?  If some FooThreads package is
> >    provided by the kernel and the pthreads implementation uses
> >    FooThreads primitives in a compatible way, the application should
> >    be okay even if it doesn't use the same interface as the gss/krb5
> >    libraries.  Should that knowledge be in the library or the
> >    application?  Should we not bother?  */
> > /* Should these be macros or enumerators?  */
> > #define FOO_THREADS_PTHREAD   1
> > #define FOO_THREADS_WIN32     2
> > #define FOO_THREADS_MACOS9    3
> > #define FOO_THREADS_MACH      4
> The library won't know which thread library and hopefully won't need to 
> know, right? It just needs to know whether it has callbacks registered and 
> is thread-safe, or it has the threading API compiled in with no callbacks 
> registered or doesn't have the threading API compiled in and is unsafe.

It won't know which version of the thread library, but it'll know at
compile time which API (if any) is compiled in as the default.

If an application doesn't want to register a lock function, it can
still do a safety check that the library wasn't built against a
different thread system.  If we assume that same API means same
implementation (which may not be true in some systems?), that should
be enough to let the application run safely.

(If it's not enough, maybe we should always make the application
register the callbacks, and always default to no locking?)

> > #define LOCKMODE_LOCK   1
> > #define LOCKMODE_UNLOCK 2
> > /* Acquire the lock.
> > 
> >    Q: Support ref-counted locks at this layer?  */
> > void fooint_lock (int mode, int locknum, const char *file, int line);
> > #define LOCK(N)    fooint_lock(LOCKMODE_LOCK,(N),__FILE__,__LINE__)
> > #define UNLOCK(N)  fooint_lock(LOCKMODE_UNLOCK,(N),__FILE__,__LINE__)
> LOCKMODE_LOCK and LOCKMODE_UNLOCK can't be private to the library, since 
> the library won't know what to pass to the locking callback to get lock 
> and unlock capability - this will be defined by the application that 
> sets the locking function.

Good point.  The macro names should perhaps be per-library too, but
the values should be consistent across all the libraries.

> I wrote a proof of concept that is rather different than what is described 
> here. I need to rework it to meet in the middle. My proof of concept has 
> the library registering the API (it knows the system that it runs on), but 
> that makes no sense because you want to avoid the overhead unless the 
> application is threaded.

Would weak references help?  Checking a function address shouldn't be
too expensive.  Or are pthread_create and friends always available
even if you don't ask for them at link time?

My hunch is that having one version of the library just do something
reasonable, as much as possible, for both threaded and non-threaded
applications, without the application having to do the registration at
all, would be ideal.  Whether it's an attainable ideal, I don't
know. :-)

> So, I just wanted to double check the advisibility of defining all of this 
> per library since that will duplicate the code per library and cause any 
> application that links to more than one library to have to set up the 
> callbacks per library (more duplicated code). (Do most apps link to only 
> one or more than one library?)

There are applications that are linking to SASL and GSS and krb5 and
krb4 and openssl and....  In theory, we can also have apps directly
using only GSS, or only SASL, and using other libraries under the
covers when using the krb5-based implementations; it certainly should
be possible.  (In the Kerberos tree itself, of course, we never have
anything quite so simple.)

I'm hoping the API is sufficient that the application only needs to
register callbacks with the libraries it uses directly; libraries
picked up indirectly should be "informed" by the directly-used library
that pulls them in.  Yes, there would be some duplicated code, but I
don't think there would be all that much.

Ken