RFC: preauth benchmarking methodology

[Nathaniel McCallum]

On Fri, 2011-06-03 at 15:48 -0400, Marcus Watts wrote:
> > Date:    Fri, 03 Jun 2011 15:07:16 EDT
> > To:      krbdev at mit.edu
> > From:    Nathaniel McCallum <npmccallum at redhat.com>
> > Subject: RFC: preauth benchmarking methodology
> > 
> > All the code referenced below comes from here:
> > https://github.com/npmccallum/krb5-anonsvn/tree/perftest/src/plugins/preauth/pe
> > rftest
> > 
> > As part of the FreeIPA project (http://freeipa.org) we are attempting to
> > add support for a variety of preauth mechanisms, such as yubikey, rsa,
> > and others.  One of the major concerns that has come up in our testing
> > is that while the current krb5 preauth mechanisms are quite quick to
> > verify, the use of external services like yubikey which may introduce
> > multi-second delays introduces scalability problems due to the
> > non-threaded, synchronous main-loop of krb5.
> > 
> > Before we attempt to fix the problem however, we need to make sure that
> > we have a standardized testing suite to measure our progress.  This
> > suite should be reusable for krb5 in other ways as well.
> > 
> > The basic idea is that we need to simulate reproducible delays in a
> > preauth plugin and measure the total responsiveness of the server when
> > these delays appear.  To this end I've created a preauth plugin called
> > 'perftest' which always approves the preauth after a delay.  The delay
> > is controlled by the name of the principal, where 1 at REALM.COM would
> > delay for 1 millisecond and 3000 at REALM.COM would delay for 3 seconds.
> > 
> > Then we measure the speeds of a set of kinit's by using the simulexec.py
> > file (in the repo above) which will execute a set of commands with a
> > given concurrency and repetition values.  The output of the script is
> > CSVs with the columns:
> >   1. Total size
> >   2. Total time (seconds)
> >   3. Parallelism
> >   4. Successes
> >   5. Failures
> >   6. Average time of successes (seconds)
> >   7. Standard deviation of #4
> > 
> > We will of course need to use a parallelism greater than the number of
> > worker processes in the kdc, or the test will be useless.
> > 
> > Does anyone have any further thoughts?
> > 
> > Nathaniel
> > 
> > _______________________________________________
> > krbdev mailing list             krbdev at mit.edu
> > https://mailman.mit.edu/mailman/listinfo/krbdev
> 
> Is your krb5kdc single threaded when it does a preauth choice?

Yes. The goal is to move to an asynchronous event loop and measure the
difference in performance.

> Older versions of krb5kdc definitely were.  If a preauth method
> took 2 seconds to execute, that's 2 seconds during which any
> other request sent in would block -- and perhaps be redirected
> to another kdc depending on how patient the client was.

That is my assumption as well.

Nathaniel