[OWW-Discuss] Problems with OWW?

[Bill F]

Bryan,

As you know, we package up all of our content every night for anyone who
wants it; we have no problem sharing. If you want to spider and do it as a
well-bahaved app, it's something we can live with. Google and Yahhoo do it a
few times per day as an example.

What was a problem was the spider from hell a few weeks back that killed
performance for a da as it repeatedly ground through OWW, killing
performance. I mean killing. The CPU and network utilization was spiked. We
could barely log into the server via SSH.

We first knocked off the server via the robots.txt file; it failed to read
the file and throtle its activity. We then banned the server via the apache
config file. When both of these methods failed, we added it to the
ipfirewall's ban list and it stopped the activity imediately.

The histogram for the month tells it all. We ate well over 100% of the
server's normal load for day.



On Tue, Apr 8, 2008 at 1:27 AM, Bryan Bishop <kanzure at gmail.com> wrote:

> On Friday 28 March 2008, Bill F wrote:
> > We were hammered today by a system spidering the OWW server out of
> > control. We worked with Rackspace into the evening to make sure that
> > the server got out from under the barrage. We ran well over twice our
> > daily bandwidth and database activity. This started last evening. The
> > system was periodically throwing database error as it started. The
> > php errors were caused when we added a new extension.
>
> I ran a spider a few months ago on OWW and did no where near this much
> damage. What's the deal?
>
> - Bryan
> ________________________________________
> http://heybryan.org/mediawiki/index.php/Roadmap
> _______________________________________________
> OpenWetWare Discussion Mailing List
> discuss at openwetware.org
> http://mailman.mit.edu/mailman/listinfo/oww-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/oww-discuss/attachments/20080408/cbd10edf/attachment.htm