Bryan,<br><br>As you know, we package up all of our content every night for anyone who wants it; we have no problem sharing. If you want to spider and do it as a well-bahaved app, it's something we can live with. Google and Yahhoo do it a few times per day as an example.<br>
<br>What was a problem was the spider from hell a few weeks back that killed performance for a da as it repeatedly ground through OWW, killing performance. I mean killing. The CPU and network utilization was spiked. We could barely log into the server via SSH. <br>
<br>We first knocked off the server via the robots.txt file; it failed to read the file and throtle its activity. We then banned the server via the apache config file. When both of these methods failed, we added it to the ipfirewall's ban list and it stopped the activity imediately.<br>
<br>The histogram for the month tells it all. We ate well over 100% of the server's normal load for day. <br><br><br><br><div class="gmail_quote">On Tue, Apr 8, 2008 at 1:27 AM, Bryan Bishop <<a href="mailto:kanzure@gmail.com">kanzure@gmail.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="Ih2E3d">On Friday 28 March 2008, Bill F wrote:<br>
> We were hammered today by a system spidering the OWW server out of<br>
> control. We worked with Rackspace into the evening to make sure that<br>
> the server got out from under the barrage. We ran well over twice our<br>
> daily bandwidth and database activity. This started last evening. The<br>
> system was periodically throwing database error as it started. The<br>
> php errors were caused when we added a new extension.<br>
<br>
</div>I ran a spider a few months ago on OWW and did no where near this much<br>
damage. What's the deal?<br>
<br>
- Bryan<br>
________________________________________<br>
<a href="http://heybryan.org/mediawiki/index.php/Roadmap" target="_blank">http://heybryan.org/mediawiki/index.php/Roadmap</a><br>
<div><div></div><div class="Wj3C7c">_______________________________________________<br>
OpenWetWare Discussion Mailing List<br>
<a href="mailto:discuss@openwetware.org">discuss@openwetware.org</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/oww-discuss" target="_blank">http://mailman.mit.edu/mailman/listinfo/oww-discuss</a><br>
</div></div></blockquote></div><br>