daily latency spike

[Russ Allbery]

"Paul B. Henson" <henson at acm.org> writes:

> We currently have two kerberos realms, each consisting of three systems
> (1 physical box and 2 virtual machines). For a while now we've been
> having an issue where once a day, almost exactly every 24 hours, the two
> physical boxes have a latency spike and don't respond to authentication
> requests, to the point where our monitoring systems alarms that they're
> down. Interestingly, this daily outage seems to happen every 24 hours
> based on when the services were last restarted.

> We see a small spike on io requests that seems to correspond with the
> delayed authentication processing. The two vm's in each realm also see
> similar spikes, but the SAN they are on has loads of iops and they don't
> seem to have an issue continuing to respond to requests during the
> spike.

> We're using openldap with the ldap backend.

I assume you've already looked for daily cron jobs?  It's common for LDAP
servers to take a nightly LDIF dump for backups, for example, which would
cause symptoms like this.

-- 
Russ Allbery (eagle at eyrie.org)              <http://www.eyrie.org/~eagle/>