KDC performance test - lookaside cache impact, testing framework

[Petr Spacek]

On 04/05/2012 10:48 PM, Chris Hecker wrote:
>
>> KDC DB in OpenLDAP (same host as KDC):
>> Without pre-authentication: 13 s, 14 s, 14s
>> With pre-authentication: 6 s, 7 s, 6 s, 7 s
>
> Are these times flipped?
Oh, yes, they are. Sorry.

Petr Spacek

>
> Chris
>
> On 2012/04/05 13:36, Petr Spacek wrote:
>> Greetings,
>>
>> my name is Petr Spacek and I work in cooperation with Red Hat on
>> Kerberos Performance Test Suite. (It's my master thesis project, some
>> details are at the end of e-mail, after the interesting part :-)
>>
>> To get familiar with KDC and Kerberized environment I did some simple
>> synthetic performance tests on Fedora's MIT KDC version 1.10.1.
>> I think results can be interesting for you.
>>
>> Test was pretty simple: 100 kinits in parallel requested 100 TGTs each
>> (for single principal), i.e. totally 10000 TGT requests. Time necessary
>> to fulfil all requests was measured.
>> These bursts were sent and measured one after another repeatedly.
>>
>> KDC was on one computer, kinits was on another computer. Detailed HW
>> configurations are not important for now, I think. Load of KDC's CPU was
>> ~ 100 %, client's load was<  30 %.
>>
>> Performance impact of some KDC configuration options was tested. Numbers
>> below are from configurations with disable_last_success = true and
>> disable_lockout = true. I can provide more details, if it will be necessary.
>>
>>
>> The results are as follows (time measured for each successive burst):
>>
>> KDC DB in local file:
>> Without pre-authentication: 9 s, 24 s, 37 s, 48 s, 40 s, 40 s
>> With pre-authentication: 26 s, 63 s, 75 s, 68 s, 72 s
>>
>> KDC DB in OpenLDAP (same host as KDC):
>> Without pre-authentication: 14 s, 36 s, 55 s, 55 s, 50 s
>> With pre-authentication: 36 s, 86 s, 83 s
>>
>> I was very surprised with there results. I repeated measurements and
>> variation between attempts was<  10 %. It's not very precise, but I
>> think it's enough to confirm observed trends.
>>
>>
>> After some time a found "KDC lookaside cache" with defined "STALE_TIME"
>> 120 s. I think it explains observed trends: Time required to fulfil one
>> burst got stable after approximately 120 s. The reason is (I think) that
>> rate of adding new requests to the cache and removing "stale" requests
>> from the cache are approximately same. When size of the cache stabilizes
>> measured times also stabilizes. (It's only hypothesis.)
>>
>>
>> Then I recompiled KDC with --disable-kdc-lookaside-cache switch and
>> repeated tests:
>>
>> KDC DB in local file:
>> Without pre-authentication: 6 s, 6 s, 6 s, 6 s
>> With pre-authentication: 8 s, 8 s, 8 s, 8 s, 8 s
>>
>> KDC DB in OpenLDAP (same host as KDC):
>> Without pre-authentication: 13 s, 14 s, 14s
>> With pre-authentication: 6 s, 7 s, 6 s, 7 s
>>
>>
>> These numbers are nicer :-D
>>
>> Please, let me ask some academic questions:
>> Why the lookaside cache is there? I looked into RFC4120. Section 3.1.2
>> http://tools.ietf.org/html/rfc4120#section-3.1.2 say:
>>
>>      Because Kerberos can run over unreliable transports such as UDP, the
>>      KDC MUST be prepared to retransmit responses in case they are lost.
>>      If a KDC receives a request identical to one it has recently
>>      processed successfully, the KDC MUST respond with a KRB_AS_REP
>>      message rather than a replay error.  In order to reduce ciphertext
>>      given to a potential attacker, KDCs MAY send the same response
>>      generated when the request was first handled.  KDCs MUST obey this
>>      replay behavior even if the actual transport in use is reliable.
>>
>> Ok, it's a standard and it has to be followed. Can be STALE_TIME lower?
>> It there a real problem, when lookaside cache is disabled? What are
>> implications?
>>
>>
>> The used tests was really synthetic and far from real situations, I know
>> that. There was only point to point network, no firewalls, only TGT
>> requests and so on.
>>
>> Now I work on general benchmarking framework. It should allow to
>> (relatively simply) write and run defined tests on bunch of machines in
>> parallel. It's aimed to end-to-end performance testing (test whole chain
>> from e.g. libpam_krb5 to KDC and back), but you can use plain kinit or
>> Kerberos library and create synthetic test. It's planed to be very general.
>>
>> Would it be interesting for real Kerberos developers? I'm open to any
>> specific requests and proposals.
>>
>> My work focuses on framework itself, but beside framework I will
>> implement real Kerberos tests. Some tests can be designed on you
>> requirements.
>>
>> Thanks for your time.