View Single Post
  #9 (permalink)  
Old 05-11-2004, 03:16 AM
ronniethedodger's Avatar
ronniethedodger ronniethedodger is offline
WebProWorld 1,000+ Club
 
Join Date: Aug 2003
Location: Central US
Posts: 1,265
ronniethedodger RepRank 1
Default

There was an apparant rash of unclaimed spiders last month. It was downright ridiculous, and nobody was owning up to them it seemed. USCity.NET shut the door on the AllTheWeb spider finally.

The one that is really confusing is the Fast Enterprise spider. This one now is a spider for hire, so to speak, from what I can tell ... and it showed up one day and went thru a page a minute which is what their FAQ says it would do. The thing was that it did that for 2-1/2 days solid. You could sit back and watch this thing on the logs methodically going thru the site ... it is by all rights a very smart link follower. Then it hit the end and started all over again, and that is when we closed the door on it.
Reply With Quote