Huge I/O wait while load testing KDC

[Ken Raeburn]

On Jan 21, 2008, at 16:52, Shivakeshav Santi wrote:
>     I have an MIT KDC installed on multiprocessor machine with  
> RHEL-4. I
> noticed two things:
>
> a) I see > 80% time spent in i/o (based on sar output)
>
>       The only major i/o would be logs and replay cache (most of  
> the i/o
> was writes). I was wondering if there was a way to turn of logging  
> and/or
> replay cache. I am looking for runtime options or any suggestions to
> improve replay cache i/o performance.

The replay cache does consume a lot of time if there are a lot of  
entries, and if you're load-testing it, there will be a lot of  
entries, as it basically lists the requests from the last N minutes.   
In most cases, however, it's not actually of significant security  
benefit to the KDC, compared with other applications.  Try giving the  
command-line option "-R none:" to disable it, if the version in  
RHEL-4 supports that.  ("-R" specifies a replay cache name; the  
"none" type that doesn't really do anything was added a while back  
for just this case.)

We've also kicked around a few ideas for improving replay cache  
performance in general, some involving changes to the on-disk format,  
some just changing how it's managed in memory, but aside from the KDC  
case we haven't had enough complaints about the performance to put  
that high on the priority list just yet.

> b) I also noticed that kerberos was using only one processor, is  
> there a
> recommended way to configure KDC on a multi processor machine so  
> that I
> could use other processors.

We have received patches for multi-threaded support in some parts of  
the KDC (mainly, allowing multiple requests to be blocked waiting for  
data from the KDC's database, which could be especially slow if it's  
in a remote LDAP server), but unfortunately I haven't had a lot of  
time to test them out so far, being busy instead with various bits of  
non-coding work.  If you'd like to try it out, and can do not just  
some regular stress testing but also exercise it with tools like  
valgrind (and especially its "helgrind" tool), the assistance would  
be appreciated, and could speed up the integration into our source  
tree.  Though if your goal is just to meet some performance criteria,  
disabling the replay cache is probably the biggest benefit.

Ken