RFC: preauth benchmarking methodology

[Nathaniel McCallum]

All the code referenced below comes from here:
https://github.com/npmccallum/krb5-anonsvn/tree/perftest/src/plugins/preauth/perftest

As part of the FreeIPA project (http://freeipa.org) we are attempting to
add support for a variety of preauth mechanisms, such as yubikey, rsa,
and others.  One of the major concerns that has come up in our testing
is that while the current krb5 preauth mechanisms are quite quick to
verify, the use of external services like yubikey which may introduce
multi-second delays introduces scalability problems due to the
non-threaded, synchronous main-loop of krb5.

Before we attempt to fix the problem however, we need to make sure that
we have a standardized testing suite to measure our progress.  This
suite should be reusable for krb5 in other ways as well.

The basic idea is that we need to simulate reproducible delays in a
preauth plugin and measure the total responsiveness of the server when
these delays appear.  To this end I've created a preauth plugin called
'perftest' which always approves the preauth after a delay.  The delay
is controlled by the name of the principal, where 1 at REALM.COM would
delay for 1 millisecond and 3000 at REALM.COM would delay for 3 seconds.

Then we measure the speeds of a set of kinit's by using the simulexec.py
file (in the repo above) which will execute a set of commands with a
given concurrency and repetition values.  The output of the script is
CSVs with the columns:
  1. Total size
  2. Total time (seconds)
  3. Parallelism
  4. Successes
  5. Failures
  6. Average time of successes (seconds)
  7. Standard deviation of #4

We will of course need to use a parallelism greater than the number of
worker processes in the kdc, or the test will be useless.

Does anyone have any further thoughts?

Nathaniel