Rate limiting Kerberos Requests

[Jack Neely]

On Tue, Sep 25, 2012 at 01:49:55PM -0700, Russ Allbery wrote:
> Jack Neely <jjneely at ncsu.edu> writes:
> 
> > Has anyone done any rate limiting or throttling of Kerberos requests?
> > I've had several situations where I had a load of 600 requests / minute
> > against my 3 kerberos slaves that caused degradation of performance for
> > everyone else.  Always from misbehaving tools or applications.
> 
> That seems like a rather slow authentication rate and low load to me.  I
> would expect a KDC to handle that level of load without breathing hard.
> What sort of configuration (hardware, software, etc.) are you using?
> 
> Our Kerberos KDCs *averaged* 2,800 requests per minute through the whole
> month of August.

Thanks for reading between the lines.  I don't have evidence that my
KDCs were overloaded, yet I got quite a few cannot reach KDC errors and
a logins stopped working everywhere.

The slaves are HP G7 blades with 12GB of RAM and a 6 core Intel Xeon.  2
servers in one DC and the other slave (and master) in the other DC.
Each DC has its own firewall/vlan for the kerberos servers.  RHEL 5
running kerb 1.6.1.

My network engineers tell me that the firewall in one DC had 8000
concurrent connections from the offending IP address to the KDCs and
4000 in the second DC.  (Oddly, the DC with only 1 slave.)  The KDCs
weren't able to handle other requests until the spike settled.

Jack
-- 
Jack Neely <jjneely at ncsu.edu>
Linux Czar, OIT Campus Linux Services
Office of Information Technology, NC State University
GPG Fingerprint: 1917 5AC1 E828 9337 7AA4  EA6B 213B 765F 3B6A 5B89