[krbdev.mit.edu #5130] Race condition in utils/support/threads.c if one thread calls exit....

[Ezra Peisach via RT]

Summary:
----------
If one runs tests/threads/prof1 without the /tmp/foo.conf or 
/tmp/foo1.conf file on an SMP machine, one observes

./prof1: prof1: ../../../src/util/support/threads.c:226: 
krb5int_getspecific: Assertion `destructors_set[keynum] == 1' failed.
Abort

every once in a while...

The problem:
--------------
One of the threads has called exit() while another thread is in  
krb5int_getspecific().

Stack trace: (one thread)

#0  0x005177a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x002a57a5 in raise () from /lib/tls/libc.so.6
#2  0x002a7209 in abort () from /lib/tls/libc.so.6
#3  0x0029ed91 in __assert_fail () from /lib/tls/libc.so.6
#4  0x00578c0a in krb5int_getspecific (keynum=K5_KEY_COM_ERR)
    at ../../../src/util/support/threads.c:226
#5  0x003c064b in get_thread_buffer ()
    at ../../../src/util/et/error_message.c:98
#6  0x003c06db in error_message (code=2)
    at ../../../src/util/et/error_message.c:146
#7  0x003c1a15 in default_com_err_proc (whoami=0xbff23b12 "./prof1", code=2,
    fmt=0x8048b03 "calling profile_init(\"%s\")", ap=0xb4dc540c 
"ç\212\004\b")
    at ../../../src/util/et/com_err.c:75
#8  0x003c1ac5 in com_err_va (whoami=0xbff23b12 "./prof1", code=2,
    fmt=0x8048b03 "calling profile_init(\"%s\")", ap=0xb4dc540c 
"ç\212\004\b")
    at ../../../src/util/et/com_err.c:104
#9  0x003c1d83 in com_err (whoami=0xbff23b12 "./prof1", code=2,
    fmt=0x8048b03 "calling profile_init(\"%s\")")
    at ../../../src/util/et/com_err.c:131
#10 0x080488f5 in worker (arg=0x0) at ../../../src/tests/threads/prof1.c:37

(another thread):
#0  0x005177a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00d911de in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#2  0x00d8de3b in _L_mutex_lock_35 () from /lib/tls/libpthread.so.0
#3  0x0057dda0 in flag_pthread_loaded () from 
/tmp/b123/lib/libkrb5support.so.0
#4  0x003c3b5c in ?? () from /tmp/b123/lib/libcom_err.so.3
#5  0x003c3c60 in et_list_lock () from /tmp/b123/lib/libcom_err.so.3
#6  0xb7fe22a0 in ?? ()
#7  0xb7fca328 in ?? ()
#8  0x003c0210 in com_err_terminate ()
    at ../../../src/util/et/error_message.c:77
#9  0x003c0210 in com_err_terminate ()
    at ../../../src/util/et/error_message.c:77
#10 0x003bfcb7 in __do_global_dtors_aux () from 
/tmp/b123/lib/libcom_err.so.3
#11 0x003c233a in _fini () from /tmp/b123/lib/libcom_err.so.3
#12 0x00523907 in _dl_fini () from /lib/ld-linux.so.2
#13 0x002a8527 in exit () from /lib/tls/libc.so.6
#14 0x08048901 in worker (arg=Could not find the frame base for "worker".
) at ../../../src/tests/threads/prof1.c:38


What is going on?
------------------

The exit handler in libcomerr calls k5_key_delete before the 
k5_mutex_destroy on com_err_hook.
k5_key_delete will set the destructors_set for the key to 0.  This is 
under a mutex_lock on the global threads lock.

While this is happening, another thread is
using the com_err library - and it wants to do a k5_getspecific which 
has a line:
assert(destructors_set[keynum] == 1) - without any mutex locking....

How to fix the problem?
-------------------------
a) user should never call exit in a thread - user should return - or use 
pthread_exit?

b) Remove the assertion on destructors_set...

c) Maybe use a mutex before testing destructors_set - and if in 
k5_getspecific - return NULL if not set - that would
indicate in the process of exiting?