Performance and Replay Cache

[Rainer Weikusat]

Steve Edgar <se10 at cornell.edu> writes:
> In the early 90s, a basic client/server protocol was developed at  
> Cornell called Cornell University Stateless Server Protocol (CUSSP).   
> Each request from the client contains a K4 ticket for  
> authentication.  The contents of each request are encrypted with  
> mk_priv.
>
> This protocol is still used on a few servers.  One of these servers  
> commonly sees an incoming request rate of 35 requests per second.   
> After converting CUSSP to use K5, the increase in latency resulting  
> from the replay cache affects performance.

There are two problems in this code: On is that it calls fsync for
every new entry, which may be nice for 'robustness' (but no recovery
procedure is implemented) but adds a fairly large, random delay to
each request that causes a new entry to be created. The other is the
garbage collector, which is supposed to do 'lazy collection'. For a
steady stream of requests, there will be a long initial build-up phase
with no garbage collection and afterwars, the collector runs in
regular and much shorter intervals and is working a lot in each run.

I had problems with this in a server applications some months ago
(excuse me if this is light on details, but I am just describing that
from memory and have been working on a lot of different things since
then). What I did was to replace the implementation (if just for not
having to read Mr Bernstein's comments anymore) with one without fsync
and with an incremental collector: After each request from the net
(which arrive in regular intervals from several hundert clients), a
call is made into the collector which removes at most a fixed, small
number of expired entries (and so the processing time is constant and
not depending on the total size of the rcache). If you have a steady
request stream, too, you could

	a) remove the sync call from krb5_rc_dfl_store: This means
           if your system 'suddenly' shuts down, you may loose parts
           of the reply cache that have not yet been written to disk.

	b) have the incoming requests drive an incremental collector

The code I use is, unfortunately, proprietary and owned by my employer
(and specificallt only implements the functionality used in this
particular application).