gss_store_cred_into() and gss_acquire_cred_from() on a client specific basis

[moore moore]

 Thank you GregI finally have it working now. By preserving the cred handles I can reuse.
This is working in multi threaded process. However with about 20 concurrent threads, there appears major lock up in the threads (all in the krb5 lib) around malloc and memory management.
krb5_is_thread_safe() returns true (1).

Is there anything to be mindful of in multi threaded context?
samples:
Thread 120 (Thread 0x2af66bc60700 (LWP 59755)):#0  __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95#1  0x00002af669d39bca in _L_lock_10381 () at malloc.c:5201#2  0x00002af669d37705 in __GI___libc_malloc (bytes=28) at malloc.c:2884#3  0x00002af668bb0001 in ?? () from /lib/x86_64-linux-gnu/libkrb5.so.3#4  0x00002af668bb057f in ?? () from /lib/x86_64-linux-gnu/libkrb5.so.3#5  0x00002af668bacdc2 in krb5_get_server_rcache () from /lib/x86_64-linux-gnu/libkrb5.so.3#6  0x00002af668ba5639 in ?? () from /lib/x86_64-linux-gnu/libkrb5.so.3#7  0x00002af668ba5cfa in krb5_rd_req_decoded () from /lib/x86_64-linux-gnu/libkrb5.so.3

Thread 121 (Thread 0x2af66b969700 (LWP 59754)):#0  __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95#1  0x00002af669d39bca in _L_lock_10381 () at malloc.c:5201#2  0x00002af669d37705 in __GI___libc_malloc (bytes=16) at malloc.c:2884#3  0x00002af668b8aeb2 in krb5_copy_data () from /lib/x86_64-linux-gnu/libkrb5.so.3#4  0x00002af668b8acc1 in ?? () from /lib/x86_64-linux-gnu/libkrb5.so.3#5  0x00002af668b909b1 in krb5_get_credentials () from /lib/x86_64-linux-gnu/libkrb5.so.3#6  0x00002af668e48a61 in get_credentials (server=<optimized out>, out_creds=<synthetic pointer>, endtime=<optimized out>, now=1559651781, cred=0x2af680902950, context=0x2af683264100) at init_sec_context.c:196#7  kg_new_connection (exts=0x2af66b958070, context=0x2af683264100, time_rec=0x2af66b9581a4, ret_flags=0x0, output_token=0x2af66b958200, actual_mech_type=0x0, input_token=0x0, input_chan_bindings=0x0, time_req=4294967295, req_flags=<optimized out>, mech_type=0x2af6690655a0 <krb5_gss_oid_array>, target_name=<optimized out>, context_handle=0x2af6841045b0, cred=0x2af680902950, minor_status=0x2af66b9581a0) at init_sec_context.c:587

Thread 122 (Thread 0x2af66beb3700 (LWP 59753)):#0  __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95#1  0x00002af669d39bca in _L_lock_10381 () at malloc.c:5201#2  0x00002af669d37705 in __GI___libc_malloc (bytes=28) at malloc.c:2884#3  0x00002af668bb0001 in ?? () from /lib/x86_64-linux-gnu/libkrb5.so.3#4  0x00002af668bb057f in ?? () from /lib/x86_64-linux-gnu/libkrb5.so.3#5  0x00002af668bacdc2 in krb5_get_server_rcache () from /lib/x86_64-linux-gnu/libkrb5.so.3#6  0x00002af668ba5639 in ?? () from /lib/x86_64-linux-gnu/libkrb5.so.3#7  0x00002af668ba5cfa in krb5_rd_req_decoded () from /lib/x86_64-linux-gnu/libkrb5.so.3#8  0x00002af668e3e210 in kg_accept_krb5 (minor_status=0x2af66bea20dc, context_handle=0x2af68052d240, verifier_cred_handle=0x2af6813def50, input_token=<optimized out>, input_chan_bindings=0x0, src_name=0x2af66bea1dc8, mech_type=mech_type at entry=0x2af66bea1dd8, output_token=output_token at entry=0x2af66bea1f20, ret_flags=ret_flags at entry=0x2af66bea1db4, time_rec=time_rec at entry=0x0, delegated_cred_handle=delegated_cred_handle at entry=0x2af66bea1dc0, exts=exts at entry=0x2af66bea1d40) at accept_sec_context.c:644#9  0x00002af668e3f5a4 in krb5_gss_accept_sec_context_ext (minor_status=0x2af66bea20dc, context_handle=0x2af68052d240, verifier_cred_handle=<optimized out>, input_token=<optimized out>, input_chan_bindings=<optimized out>, src_name=<optimized out>, mech_type=mech_type at entry=0x2af66bea1dd8, output_token=output_token at entry=0x2af66bea1f20, ret_flags=ret_flags at entry=0x2af66bea1db4, time_rec=time_rec at entry=0x0, delegated_cred_handle=delegated_cred_handle at entry=0x2af66bea1dc0, exts=exts at entry=0x2af66bea1d40) at accept_sec_context.c:1308

Thread 123 (Thread 0x2af66b840700 (LWP 59752)):#0  __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95#1  0x00002af669d39bca in _L_lock_10381 () at malloc.c:5201#2  0x00002af669d37705 in __GI___libc_malloc (bytes=24) at malloc.c:2884#3  0x00002af668bb9f34 in ?? () from /lib/x86_64-linux-gnu/libkrb5.so.3#4  0x00002af668bba10a in k5_os_init_context () from /lib/x86_64-linux-gnu/libkrb5.so.3#5  0x00002af668b9689f in krb5_init_context_profile () from /lib/x86_64-linux-gnu/libkrb5.so.3#6  0x00002af668e51086 in krb5_gss_release_name (minor_status=0x2af66b82f104, input_name=0x2af67f4900b0) at rel_name.c:34#7  0x00002af668e32fb8 in gssint_release_internal_name (minor_status=minor_status at entry=0x2af66b82f104, mech_type=<optimized out>, internal_name=internal_name at entry=0x2af67f4900b0) at g_glue.c:577#8  0x00002af668e3b007 in gss_release_name (minor_status=minor_status at entry=0x2af66b82f104, input_name=input_name at entry=0x2af66b82f108) at g_rel_name.c:78
    On Friday 24 May 2019, 19:07:25 GMT+1, Greg Hudson <ghudson at mit.edu> wrote:  
 
 On 5/24/19 12:19 PM, moore moore wrote:
> gss_acquire_cred_from() for gss_name:u:clientuser1 at TEST.COM, 
>  store:0x6ddf68, elem:0x6ddf78
> 
> [28197] 1558713187.312227: Retrieving clientuser1 at TEST.COM from
> FILE:/krb5/dest/var/krb5/user/0/client.keytab (vno 0, enctype 0) with
> result: 2/Key table file '/krb5/dest/var/krb5/user/0/client.keytab' not
> found

Oh, I see, it's the only trace log message.  It seems we don't generate
any trace logs when scanning the ccache, so there's nothing in the trace
log to verify what ccache gss_acquire_cred_from() is trying to use.

Since the trace logs aren't helpful in this case, I would suggest adding
debugging code to verify what values are contained within store, not
just that it has the same pointer value.  If you're in a position to
step through the libgssapi_krb5 code, you could alternatively see what's
happening inside scan_ccache().

Separately, it occurs to me that if the same process is handling the
reauths, you could just save the GSS cred handle and not bother with
gss_store_cred_into/gss_acquire_cred_from.  You'd have to create your
own table mapping client names to GSS cred handles to make this work,
but there would be fewer GSS and krb5 operations to worry about.  Just
another option.