[OWW-Discuss] OWW and yacy.net: yacybot is killing performance

[Bill F]

Does Anyone know anything about a spider bot called "yacybot":

http://www.yacy.net

It's a German host where you download the code from.

They claim to be affiliated with the Karlsrueh Institute of Technology.
They're really killing our database performace.

The IP address used in the request is invalid. Since this is the case, this
may be a denial-of-service attack. I was unable to access the system for
over a minute a few hours ago. The database has been timing out because of
the level of activity.

One indicator of how intrusive they are is that when I search for them, I'm
coming up with logfile entries for hosts all over the net that this bot is
also hitting.

This caims to be a peer-to-peer search engine.

They're now in the 'deny 'list in robots.txt. They do appear to read the
file when they start. If I see them continuing to hit us, we'll take the
next step and treat any system running this bot as a spammer.

Before we take further measure to throttle them, I just want to know if
anyone has experience with them.

Thanks.

Bill
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/oww-discuss/attachments/20080327/beaf76a1/attachment.htm