is krb5_cc_initialize() thread safe

[Olga Kornievskaia]

On Thu, Feb 20, 2025 at 6:25 PM Greg Hudson <ghudson at mit.edu> wrote:
>
> On 2/20/25 16:47, Olga Kornievskaia wrote:
> > I haven't been able to find in documentation whether or not
> > krb5_cc_initialize() is thread safe? From a practical point of view, I
> > believe it is not. But a call to "krb5_is_thread_safe() returns true".
>
> krb5_is_thread_safe() just reports whether the library was compiled with
> mutex calls.
>
> krb5_cc_initialize() is thread-safe to the extent that it shouldn't
> exhibit memory errors if called in multiple threads with the same
> krb5_ccache (and different krb5_contexts, as krb5_context is not a
> thread-protected object).  It is inherently not very concurrency-safe,
> whether the concurrency is due to threads or processes, because at the
> end of the call the cache is empty (no TGT) and therefore not usable to
> obtain service tickets.  (Well, "inherently" according to the
> traditional semantics.  Heimdal has adopted a behavior change to
> krb5_cc_initialize() where initializing a cache handle isn't externally
> visible until the first non-config credential is stored.  We didn't go
> that route.)
>
> We did add a way in 1.20 to atomically replace a credentials cache of
> most types: krb5_cc_move() a MEMORY cache onto the target.  This method
> is used automatically by gss_store_cred()/gss_store_cred_into(), and by
> krb5_get_init_creds_*() when an output cache is specified using
> krb5_get_init_creds_opt_set_out_ccache().  The commit that added this
> behavior is:
>
> https://github.com/krb5/krb5/commit/371f09d4bf4ca0c7ba15c5ef909bc35307ed9cc3
>
> > In NFS's gssd implementation, while trying to refresh credentials, the
> > code tries to store creds and as part of calls into
> > krb5_cc_initialize() to initial credential cache (of either FILE or
> > MEMORY type). We've observed that if two threads try to update the
> > same cache and thus call into krb5_cc_initialize(), one succeeds and
> > the other fails with "Internal credentials cache error". Is that
> > unexpected?
>
> In this scenario the two threads a probably using separate krb5_ccache
> handles to perform the initialize and store operations, so libkrb5
> locking the handles doesn't prevent the operations from trying to do
> their work at the same time--the same as if two different processes
> running two different programs were trying to initialize and store to
> FILE caches with the same filename.  I can see one order of
> sub-operations which would generate that error message, right off the
> bat: we start by unlink()ing the file and then opening it with O_EXCL.
> If these operations are interleaved (T1 unlink, T2 unlink, T1 open, T2
> open), the second open() will return an EEXIST error, which is
> translated to KRB5_FCC_INTERNAL.

Does this sound like there might be a (solvable) problem in the
krb5_cc_initiatize() that can guarantee that multiple threads can call
it simultaneously (on as you noted their own pointer of the
krb5_ccache structure that they have gotten from doing
krb5_cc_resolve() and both threads would get a non-error return from
the function? Or is that the caller's (like gssd) responsibility not
call them concurrently?

And I'm also wondering what about other krb5_* functions are they
"safe" under concurrency or should gssd be doing any krb* api call
under a lock?