KDC performance test - lookaside cache impact, testing framework

[Petr Spacek]

Greetings,

my name is Petr Spacek and I work in cooperation with Red Hat on 
Kerberos Performance Test Suite. (It's my master thesis project, some 
details are at the end of e-mail, after the interesting part :-)

To get familiar with KDC and Kerberized environment I did some simple 
synthetic performance tests on Fedora's MIT KDC version 1.10.1.
I think results can be interesting for you.

Test was pretty simple: 100 kinits in parallel requested 100 TGTs each 
(for single principal), i.e. totally 10000 TGT requests. Time necessary 
to fulfil all requests was measured.
These bursts were sent and measured one after another repeatedly.

KDC was on one computer, kinits was on another computer. Detailed HW 
configurations are not important for now, I think. Load of KDC's CPU was 
~ 100 %, client's load was < 30 %.

Performance impact of some KDC configuration options was tested. Numbers 
below are from configurations with disable_last_success = true and 
disable_lockout = true. I can provide more details, if it will be necessary.


The results are as follows (time measured for each successive burst):

KDC DB in local file:
Without pre-authentication: 9 s, 24 s, 37 s, 48 s, 40 s, 40 s
With pre-authentication: 26 s, 63 s, 75 s, 68 s, 72 s

KDC DB in OpenLDAP (same host as KDC):
Without pre-authentication: 14 s, 36 s, 55 s, 55 s, 50 s
With pre-authentication: 36 s, 86 s, 83 s

I was very surprised with there results. I repeated measurements and 
variation between attempts was < 10 %. It's not very precise, but I 
think it's enough to confirm observed trends.


After some time a found "KDC lookaside cache" with defined "STALE_TIME" 
120 s. I think it explains observed trends: Time required to fulfil one 
burst got stable after approximately 120 s. The reason is (I think) that 
rate of adding new requests to the cache and removing "stale" requests 
from the cache are approximately same. When size of the cache stabilizes 
measured times also stabilizes. (It's only hypothesis.)


Then I recompiled KDC with --disable-kdc-lookaside-cache switch and 
repeated tests:

KDC DB in local file:
Without pre-authentication: 6 s, 6 s, 6 s, 6 s
With pre-authentication: 8 s, 8 s, 8 s, 8 s, 8 s

KDC DB in OpenLDAP (same host as KDC):
Without pre-authentication: 13 s, 14 s, 14s
With pre-authentication: 6 s, 7 s, 6 s, 7 s


These numbers are nicer :-D

Please, let me ask some academic questions:
Why the lookaside cache is there? I looked into RFC4120. Section 3.1.2 
http://tools.ietf.org/html/rfc4120#section-3.1.2 say:

    Because Kerberos can run over unreliable transports such as UDP, the
    KDC MUST be prepared to retransmit responses in case they are lost.
    If a KDC receives a request identical to one it has recently
    processed successfully, the KDC MUST respond with a KRB_AS_REP
    message rather than a replay error.  In order to reduce ciphertext
    given to a potential attacker, KDCs MAY send the same response
    generated when the request was first handled.  KDCs MUST obey this
    replay behavior even if the actual transport in use is reliable.

Ok, it's a standard and it has to be followed. Can be STALE_TIME lower? 
It there a real problem, when lookaside cache is disabled? What are 
implications?


The used tests was really synthetic and far from real situations, I know 
that. There was only point to point network, no firewalls, only TGT 
requests and so on.

Now I work on general benchmarking framework. It should allow to 
(relatively simply) write and run defined tests on bunch of machines in 
parallel. It's aimed to end-to-end performance testing (test whole chain 
from e.g. libpam_krb5 to KDC and back), but you can use plain kinit or 
Kerberos library and create synthetic test. It's planed to be very general.

Would it be interesting for real Kerberos developers? I'm open to any 
specific requests and proposals.

My work focuses on framework itself, but beside framework I will 
implement real Kerberos tests. Some tests can be designed on you 
requirements.

Thanks for your time.

-- 
Petr Spacek  @  Red Hat  @  Brno office