RFC: preauth benchmarking methodology

[Marcus Watts]

> Date:    Fri, 03 Jun 2011 15:50:27 EDT
> To:      Marcus Watts <mdw at umich.edu>
> cc:      krbdev at mit.edu
> From:    Nathaniel McCallum <npmccallum at redhat.com>
> Subject: Re: RFC: preauth benchmarking methodology
> 
> On Fri, 2011-06-03 at 15:48 -0400, Marcus Watts wrote:
> > > Date:    Fri, 03 Jun 2011 15:07:16 EDT
> > > To:      krbdev at mit.edu
> > > From:    Nathaniel McCallum <npmccallum at redhat.com>
> > > Subject: RFC: preauth benchmarking methodology
> > > 
> > > All the code referenced below comes from here:
> > > https://github.com/npmccallum/krb5-anonsvn/tree/perftest/src/plugins/preaut
> h/pe
> > > rftest
> > > 
> > > As part of the FreeIPA project (http://freeipa.org) we are attempting to
> > > add support for a variety of preauth mechanisms, such as yubikey, rsa,
> > > and others.  One of the major concerns that has come up in our testing
> > > is that while the current krb5 preauth mechanisms are quite quick to
> > > verify, the use of external services like yubikey which may introduce
> > > multi-second delays introduces scalability problems due to the
> > > non-threaded, synchronous main-loop of krb5.
> > > 
> > > Before we attempt to fix the problem however, we need to make sure that
> > > we have a standardized testing suite to measure our progress.  This
> > > suite should be reusable for krb5 in other ways as well.
> > > 
> > > The basic idea is that we need to simulate reproducible delays in a
> > > preauth plugin and measure the total responsiveness of the server when
> > > these delays appear.  To this end I've created a preauth plugin called
> > > 'perftest' which always approves the preauth after a delay.  The delay
> > > is controlled by the name of the principal, where 1 at REALM.COM would
> > > delay for 1 millisecond and 3000 at REALM.COM would delay for 3 seconds.
> > > 
> > > Then we measure the speeds of a set of kinit's by using the simulexec.py
> > > file (in the repo above) which will execute a set of commands with a
> > > given concurrency and repetition values.  The output of the script is
> > > CSVs with the columns:
> > >   1. Total size
> > >   2. Total time (seconds)
> > >   3. Parallelism
> > >   4. Successes
> > >   5. Failures
> > >   6. Average time of successes (seconds)
> > >   7. Standard deviation of #4
> > > 
> > > We will of course need to use a parallelism greater than the number of
> > > worker processes in the kdc, or the test will be useless.
> > > 
> > > Does anyone have any further thoughts?
> > > 
> > > Nathaniel
> > > 
> > > _______________________________________________
> > > krbdev mailing list             krbdev at mit.edu
> > > https://mailman.mit.edu/mailman/listinfo/krbdev
> > 
> > Is your krb5kdc single threaded when it does a preauth choice?
> 
> Yes. The goal is to move to an asynchronous event loop and measure the
> difference in performance.
> 
> > Older versions of krb5kdc definitely were.  If a preauth method
> > took 2 seconds to execute, that's 2 seconds during which any
> > other request sent in would block -- and perhaps be redirected
> > to another kdc depending on how patient the client was.
> 
> That is my assumption as well.
> 
> Nathaniel
> 

Sounds to me that initially you should expect the number of
worker processes to be = to the number of kdc machines.  So,
w/ 1 machine, any degree of parallelism will give what
you define as "non-useless" numbers.

The deviation of the # of successes -- interesting but is that the only
deviation you want?  I suspect the distribution of times for successes
will be interesting, not just the average or even the deviation.

Are you using a local db, or ldap?  If you have more than one
kdc, are they serving requests randomly or is there a preferred
order, and where is the master in that order?

Of course, in any "real" environment, kinit's aren't sequentially
issued; they're issued "at will" independently by a large number
of independent machines.  So in terms of your test, that means
"parallelism" is not an integer and actually a function of demand,
execution time, % of success, and perhaps some measure of
resource contention.  In a larger environment, or even more an
environment that updates success / failure, update contention
will also be important.  Incremental replication may further
complicate matters.

When designing your parallel architecture, you should probably
also consider the possibility of an intentional DOS attack.

Mostly out of curiosity; are you using vm's for this, or real machines?

				-Marcus Watts