improving kadmind change-password performance

[Greg Hudson]

On 11/11/2012 04:40 PM, Danny Thomas wrote:
> kadmind hits 100% CPU when load-testing with <100 simulated clients.

For password changes, kadmind has to run the string-to-key algorithm on
the new password for each enctype in supported_enctypes (which defaults
to AES-256, AES-128, DES3, and RC4).  The string-to-key algorithm for
the AES enctypes is deliberately slow in order to make dictionary
attacks harder.  I believe this operation is swamping any other
performance bottlenecks.

> I tend to think the CPU comes from crypto operations and there'd be a
> big benefit from spreading the load across cores.

Our kadmind is single-threaded, and we don't currently have any support
for multiple worker processes or client-side selection between multiple
kadmind servers.

I can't think of any reason why you couldn't run multiple kadmind
processes for the same realm on different ports, but it would be
entirely manual; each kadmind process would need its own config file,
and the front-end would need to choose between multiple client config
files in order to select a particular kadmind port.

> I saw a commit "Use blocking locks in krb5kdc and libkadm5srv"
> mentioned on the mailing list. Would that help ?

I doubt it will help if you're pegging the CPU.  It might help with
other problems which might arise under load.  It would probably be
required in order to make multiple kadmind processes play well together
under load.

> Finally, kadmind is only accessible from a few of our servers so is
> there any opportunity to use faster but weaker crypto methods for a
> service not exposed to end-users ?

The crypto operations used to protect the password change operation are
probably not the performance bottleneck; instead, it's the operations
used to turn the password into a set of keys.  Since those keys will be
used by the KDC as well as kadmind, you probably don't want to use
weaker enctypes.

However, you could reduce the list of enctypes used in
supported_enctypes in order to reduce the number of string-to-key
operations kadmind has to perform.  The default value is:

aes256-cts-hmac-sha1-96:normal aes128-cts-hmac-sha1-96:normal
des3-cbc-sha1:normal arcfour-hmac-md5:normal

You can reduce it from there.  supported_enctypes should be placed in a
[realms] subsection of kdc.conf (or krb5.conf if you don't use kdc.conf).

It's also possible, in theory, to shift the burden of the string-to-key
operations to the client using the setkey RPC.  But there's no
command-line interface to this RPC, and the documentation for
Authen::Krb5::Admin doesn't suggest that it provides a binding for it.