[krbdev.mit.edu #8918] KDC and kadmind fork with DB open, breaking LMDB KDB module

[Greg Hudson via RT]

Thu Jun 18 21:18:31 2020: Request 8918 was acted upon.
 Transaction: Ticket created by ghudson at mit.edu
       Queue: krb5
     Subject: KDC and kadmind fork with DB open, breaking LMDB KDB module
       Owner: Nobody
  Requestors: ghudson at mit.edu
      Status: open
 Ticket <URL: https://krbdev.mit.edu/rt/Ticket/Display.html?id=8918 >


lmdb.h says "Use an MDB_env* in the process which opened it, not after fork()."
The reason is that LMDB tracks stale reader entries by pid. (Rather than using
kill() to detect if the locking pid corresponds to an active process, it
creates a POSIX one-byte range lock using the pid as an offset when the DB is
opened, and tests for the existence of the lock. A fork()/exit() causes this
lock to be dropped, even though the environment fd is still open.)

krb5kdc and kadmind both open the database before daemonizing, so they violate
this restriction. The consequence is that their reader entries can be stolen
and invalidated by other processes (including each other). The bereft process
will then receive MDB_BAD_RSLOT ("Invalid reuse of reader locktable slot")
errors when trying to renew its read transaction.

This bug does not manifest in the automated tests, because we mostly start
krb5kdc and kadmind with the nofork option set. It also may not manifest if
krb5kdc and kadmind are started simultaneously, as they may both claim their
reader transaction slots before either one of them forks and exits.

My manual process for reproducing the bug is:

make testrealm
pkill krb5kdc
pkill kadmind
krb5kdc
kadmind
kinit user

See also: https://mailman.mit.edu/pipermail/kerberos/2020-June/022507.html