Fwd: Need help to recover from database corruption

[June Newman]

I've been informed that mail to my other address will be blocked.  Please
respond to this mail, and I should receive it.  I apologize for the spam
and inconvenience and thank you for your assistance.




---------- Forwarded message ----------
From: June Newman <june.newman at teamaol.com>
Date: Fri, Nov 18, 2016 at 10:16 AM
Subject: Need help to recover from database corruption
To: kerberos at mit.edu


We have a long running enterprise kerberos system that appears to have
corruption.  Our KDCs are running CentOS 6.8 and we have the latest kerb
implementation for Cent 6.

Yesterday morning, we starting seeing problems with some users not being
able to authenticate, and as the morning progressed the slave KDC processes
started crashing.  On restart krb5kdc would crash within minutes.

We identified certain problem principals after evaluating core dumps, and
the principals were created over a period of weeks.  Running getprinc
against those principals consistently causes kadmin.local to crash, while a
getprinc in kadmin consistently reports that the principals don't exist.
If we run kdb5_util to dump the database we don't see the principals in the
ASCII dump file, but if we run strings agains the principal file and grep
for the principals they're found.

We've tried to work around the corrupt principals by running 'kdb5_util
dump -recurse' and 'kdb5_util dump -rev'  but it has made no difference in
the dump file.

Does anyone have advice on how we can recover the database?  We are working
in parallel to rebuild from an older backup, but it would be ideal if we
could recover the more complete database.


Regards,
June