make check fails on AIX 5.3
Gavin Sherry
swm at alcove.com.au
Tue Aug 18 11:49:43 EDT 2009
2009/8/15 Ken Raeburn <raeburn at mit.edu>
> On Aug 14, 2009, at 18:28, Gavin Sherry wrote:
>
>> The issue is that com_err_initialize() is being called more than once. The
>> first time, it is done via krb5int_initialize_library(), the second via
>> the
>> bt above. This should be guarded by pthread_once(), but that's not working
>> (although, I've verified that in a trivial program pthread_once() is
>> behaving as expected).
>>
>
> Strange.... We do definitely depend on pthread_once behaving itself.
>
Yes, it seems like a reasonable expectation :).
>
>
> Here's some more debugging info. Before the first call,
>> com_err_initialize__once is set to:
>>
>> $2 = {once = {__on_word = {0, 0, 0, 0, 0, 0, 0, 2, 0 <repeats 20 times>}},
>> error = 0, did_run = 0, fn = @0xf08d61dc: 0xd21f3eb4
>> <com_err_initialize__aux>}
>>
>> After the first call:
>>
>> $4 = {once = {__on_word = {1, 0, 0, 0, 0, 0, 0, 9, 0 <repeats 20 times>}},
>> error = 0, did_run = 1, fn = @0xf08d61dc: 0xd21f3eb4
>> <com_err_initialize__aux>}
>>
>> Before the second call:
>>
>> $5 = {once = {__on_word = {2, 0, 0, 0, 0, 0, 270008407, 9, 1, 0 <repeats
>> 19
>> times>}}, error = 0, did_run = 1, fn = @0xf08df2dc: 0xd21eefb4
>> <com_err_initialize__aux>}
>>
>
> I'm not 100% sure when you say "call" which function you're referring to.
> Is "after the first call" after com_err_initialize__aux returns but
> pthread_once is still in progress, or after pthread_once returns?
I'll make this clearer:
Here's the value before pthread_once() is invoked:
$1 = {once = {__on_word = {0, 0, 0, 0, 0, 0, 270246039, 9, 1, 0 <repeats 19
times>}}, error = 0, did_run = 0,
fn = @0xf08d61dc: 0xd21f3eb4 <com_err_initialize__aux>}
Once pthread_once() is invoked, this is changed to:
$2 = {once = {__on_word = {2, 0, 0, 0, 0, 0, 270246039, 9, 1, 0 <repeats 19
times>}}, error = 0, did_run = 0,
fn = @0xf08d61dc: 0xd21f3eb4 <com_err_initialize__aux>}
At the end of pthread_once():
$3 = {once = {__on_word = {1, 0, 0, 0, 0, 0, 270246039, 9, 1, 0 <repeats 19
times>}}, error = 0, did_run = 1,
fn = @0xf08d61dc: 0xd21f3eb4 <com_err_initialize__aux>}
Then, when com_err_initialize__aux() is invoked again:
$4 = {once = {__on_word = {2, 0, 0, 0, 0, 0, 270246039, 9, 1, 0 <repeats 19
times>}}, error = 0, did_run = 0,
fn = @0xf08df2dc: 0xd21eefb4 <com_err_initialize__aux>}
It does not break inside pthread_once() here, when it detects the data
change. I think your theory looks correct. At the first call to
com_err_initialize__aux:
(gdb) print &com_err_initialize__once
$6 = (k5_init_t *) 0xf08d600c
And the second time:
(gdb) print &com_err_initialize__once
$7 = (k5_init_t *) 0xf08df10c
Oh, and this:
(gdb) break error_message
Breakpoint 4 at 0xd21f48d0: file error_message.c, line 121. (2 locations)
(gdb) info watchpoints
Num Type Disp Enb Address What
1 breakpoint keep y <MULTIPLE> 0xd21f3ec8
breakpoint already hit 1 time
1.1 y 0xd21f3ec8 in com_err_initialize__aux at
error_message.c:126
1.2 y 0xd21eefc8 in com_err_initialize__aux at
error_message.c:126
...
So, it seems that the libraries are loaded twice. I'm not sure how to fix
this though.
>
>
> It's also very interesting to me, though, that the "fn" field changes value
> in between $4 and $5, but it still seems to refer to the same function
> address.
Excellent point.
> Is it possible two copies of the library have been loaded at different
> addresses, with different com_err_initialize__* values, but sharing a common
> libkrb5support library with its internal thread-support data? If that's so,
> I could imagine gdb showing you different locations for "print
> com_err_initialize__once" depending on the current execution context. Try
> printing the addresses of these symbols in each context, and/or using "info
> sharedlibrary" in gdb.
>
Right. So, the first time com_err_initialize__aux() (stopped at the that
function):
(gdb) info sharedlibrary
Text Range Data Range Syms Shared Object
Library
0xd0142124-0xd0145cee 0xf047d000-0xf04bf4c8 Yes
/usr/lib/libpthreads.a(shr_comm.o)
0xd0961d80-0xd0aead7f 0xf04c44c8-0xf0532638 Yes
/usr/lib/librtl.a(shr.o)
0xd21f8150-0xd2200a6b 0xf08d0bc0-0xf08d1284 Yes
/home/build/gsherry/tools/krb5/1.6.2/dist/aix5_ppc_32/lib/libkrb5support.so
0xd21f3150-0xd21f7021 0xf08d5f70-0xf08d63c8 Yes
/home/build/gsherry/tools/krb5/1.6.2/dist/aix5_ppc_32/lib/libcom_err.so
0xd246b150-0xd24acaac 0xf08d2f46-0xf08d4740 Yes
/home/build/gsherry/tools/krb5/1.6.2/dist/aix5_ppc_32/lib/libk5crypto.so
0xd0274180-0xd02a4ccb 0xf020f000-0xf0213048 Yes
/usr/lib/libpthreads.a(shr.o)
0xd014121c-0xd014193e 0xf0427508-0xf0427630 Yes
/usr/lib/libcrypt.a(shr.o)
0xd21ee250-0xd21f2121 0xf08df070-0xf08df4c8 Yes
/home/build/gsherry/tools/krb5/1.6.2/dist/aix5_ppc_32/lib/libcom_err.a(libcom_err.so)
0xd21e52d0-0xd21edbeb 0xf08ddd40-0xf08de404 Yes
/home/build/gsherry/tools/krb5/1.6.2/dist/aix5_ppc_32/lib/libkrb5support.a(libkrb5support.so)
0xd238b250-0xd246a4b6 0xf08d7556-0xf08dc540 Yes
/home/build/gsherry/tools/krb5/1.6.2/dist/aix5_ppc_32/lib/libkrb5.a(libkrb5.so)
0xd019721c-0xd01988ae 0xf02140f8-0xf02140f8 Yes
/usr/lib/libpthreads_compat.a(shr.o)
0xd0336500-0xd05e390d 0xf037eb40-0xf04266d8 Yes
/usr/lib/libc.a(shr.o)
Second time we call the function:
(gdb) info sharedlibrary
Text Range Data Range Syms Shared Object
Library
0xd0142124-0xd0145cee 0xf047d000-0xf04bf4c8 Yes
/usr/lib/libpthreads.a(shr_comm.o)
0xd0961d80-0xd0aead7f 0xf04c44c8-0xf0532638 Yes
/usr/lib/librtl.a(shr.o)
0xd21f8150-0xd2200a6b 0xf08d0bc0-0xf08d1284 Yes
/home/build/gsherry/tools/krb5/1.6.2/dist/aix5_ppc_32/lib/libkrb5support.so
0xd21f3150-0xd21f7021 0xf08d5f70-0xf08d63c8 Yes
/home/build/gsherry/tools/krb5/1.6.2/dist/aix5_ppc_32/lib/libcom_err.so
0xd246b150-0xd24acaac 0xf08d2f46-0xf08d4740 Yes
/home/build/gsherry/tools/krb5/1.6.2/dist/aix5_ppc_32/lib/libk5crypto.so
0xd0274180-0xd02a4ccb 0xf020f000-0xf0213048 Yes
/usr/lib/libpthreads.a(shr.o)
0xd014121c-0xd014193e 0xf0427508-0xf0427630 Yes
/usr/lib/libcrypt.a(shr.o)
0xd21ee250-0xd21f2121 0xf08df070-0xf08df4c8 Yes
/home/build/gsherry/tools/krb5/1.6.2/dist/aix5_ppc_32/lib/libcom_err.a(libcom_err.so)
0xd21e52d0-0xd21edbeb 0xf08ddd40-0xf08de404 Yes
/home/build/gsherry/tools/krb5/1.6.2/dist/aix5_ppc_32/lib/libkrb5support.a(libkrb5support.so)
0xd238b250-0xd246a4b6 0xf08d7556-0xf08dc540 Yes
/home/build/gsherry/tools/krb5/1.6.2/dist/aix5_ppc_32/lib/libkrb5.a(libkrb5.so)
0xd019721c-0xd01988ae 0xf02140f8-0xf02140f8 Yes
/usr/lib/libpthreads_compat.a(shr.o)
0xd0336500-0xd05e390d 0xf037eb40-0xf04266d8 Yes
/usr/lib/libc.a(shr.o)
>
> Getting only one copy of the krb5 library (and friends) is also kind of
> assumed in our code -- or, at least, if you get multiple copies, you get
> multiple copies of the whole set with symbolic references resolved within
> each set. We may or may not be setting all the right flags to encourage the
> OS to make the latter happen....
Some of the krb5 libraries seem to be duplicated above.
>
>
> Now, I set a watch point in the debugging to stop when this structure is
>> changed. GDB does not break between the first and second calls. This might
>> be a short coming of GDB, I'm not sure.
>>
>
> Yeah, there are sometimes environments or cases where it doesn't seem to
> work well. Make sure you report that, especially if you can narrow down a
> simple test case. If your watch expression is something like
> "com_err_initialize__once.once.__on_word[0]", and the problem really is
> loading two libraries (or copies of the same library) that define
> com_err_initialize__once, it's an interesting question whether the
> watchpoint should trip as a result of the execution location (program
> counter) changing so that gdb would find one version of the variable instead
> of the other. But if it is the same location the whole time, and it changes
> without the watchpoint tripping, it sounds like a gdb bug.
>
I'll investigate that.
>
> Ken
>
Thanks,
Gavin
More information about the krbdev
mailing list