make check fails on AIX 5.3

[Ken Raeburn]

On Aug 14, 2009, at 18:28, Gavin Sherry wrote:
> The issue is that com_err_initialize() is being called more than  
> once. The
> first time, it is done via krb5int_initialize_library(), the second  
> via the
> bt above. This should be guarded by pthread_once(), but that's not  
> working
> (although, I've verified that in a trivial program pthread_once() is
> behaving as expected).

Strange....  We do definitely depend on pthread_once behaving itself.

> Here's some more debugging info. Before the first call,
> com_err_initialize__once is set to:
>
> $2 = {once = {__on_word = {0, 0, 0, 0, 0, 0, 0, 2, 0 <repeats 20  
> times>}},
> error = 0, did_run = 0, fn = @0xf08d61dc: 0xd21f3eb4
> <com_err_initialize__aux>}
>
> After the first call:
>
> $4 = {once = {__on_word = {1, 0, 0, 0, 0, 0, 0, 9, 0 <repeats 20  
> times>}},
> error = 0, did_run = 1, fn = @0xf08d61dc: 0xd21f3eb4
> <com_err_initialize__aux>}
>
> Before the second call:
>
> $5 = {once = {__on_word = {2, 0, 0, 0, 0, 0, 270008407, 9, 1, 0  
> <repeats 19
> times>}}, error = 0, did_run = 1, fn = @0xf08df2dc: 0xd21eefb4
> <com_err_initialize__aux>}

I'm not 100% sure when you say "call" which function you're referring  
to.  Is "after the first call" after com_err_initialize__aux returns  
but pthread_once is still in progress, or after pthread_once returns?

It's also very interesting to me, though, that the "fn" field changes  
value in between $4 and $5, but it still seems to refer to the same  
function address.  Is it possible two copies of the library have been  
loaded at different addresses, with different com_err_initialize__*  
values, but sharing a common libkrb5support library with its internal  
thread-support data?  If that's so, I could imagine gdb showing you  
different locations for "print com_err_initialize__once" depending on  
the current execution context.  Try printing the addresses of these  
symbols in each context, and/or using "info sharedlibrary" in gdb.

Getting only one copy of the krb5 library (and friends) is also kind  
of assumed in our code -- or, at least, if you get multiple copies,  
you get multiple copies of the whole set with symbolic references  
resolved within each set.  We may or may not be setting all the right  
flags to encourage the OS to make the latter happen....

> Now, I set a watch point in the debugging to stop when this  
> structure is
> changed. GDB does not break between the first and second calls. This  
> might
> be a short coming of GDB, I'm not sure.

Yeah, there are sometimes environments or cases where it doesn't seem  
to work well.  Make sure you report that, especially if you can narrow  
down a simple test case.  If your watch expression is something like  
"com_err_initialize__once.once.__on_word[0]", and the problem really  
is loading two libraries (or copies of the same library) that define  
com_err_initialize__once, it's an interesting question whether the  
watchpoint should trip as a result of the execution location (program  
counter) changing so that gdb would find one version of the variable  
instead of the other.  But if it is the same location the whole time,  
and it changes without the watchpoint tripping, it sounds like a gdb  
bug.

Ken