Re: Search Bots Eating Bandwidth
I was having an issue with this a couple years ago... Altavista & Lycos spiders ate up our bandwith with a vengence.
We blocked the spiders IPs, (about 50 all together), and 2 weeks later we were dropped from Altavista and Lycos.
Mind you, no one would care today, but 6-7 years ago Altavista & Lycos were HUGE.
I know bandwith is expensive when you don't host your own sites from a datacenter or the like, but if spiders are crawling the crap out of your site, it is for a reason.
I let them go.
Last edited by timmathews.com; 06-26-2007 at 06:54 PM.
Reason: more explanation
|