View Single Post
  #11 (permalink)  
Old 06-26-2007, 09:12 PM
zbatia's Avatar
zbatia zbatia is offline
WebProWorld Pro
 
Join Date: Jul 2003
Location: Baltimore, MD
Posts: 130
zbatia RepRank 1
Default Re: Search Bots Eating Bandwidth

I just had to pay recently for extra bandwidth usage as well. I don't think it's because of bots (and I see them regularly, especially Yahoo's) but because of amount of a spam. I was sick and tired of cleaning up thousands(!) of spam e-mails and not only e-mails. The Bulletin Board was suffering from spam messages originated by the special bots for phpbb.

I have decided to gather the statistics who are the biggest spammers. For one month I have checked every spam e-mail and I wrote down the headers' IPs. Then, I blocked the whole range of IPs even down to A-class addresses. Believe or not, I have slashed spam at ~80%!

I was so angry, that I have overestimated my results and mistakenly slashed 65.x IP range where the Google's bot lives. You can imagine what happened to the web site with a PR 6...
The number 6 turned to 0! Our sales stopped completely for 2 months. It took me awhile (many hours of hard work and web site optimization) to get back on track.

The positive thing is that I am working in the right direction now by allowing the Ip addresses to hit my site only from the areas I want. It's like the top-bottom approach to the security.
If you want, get my results file here (some people asked me to post it):
http://www.800-security.com/tech/SPAMaddresses.txt
Please be careful, and verify your restrictions. The biggest spammers are Poland, Russia, and Asian region. There are some in America, as well.
Use the following site to check the WHOIS, etc services:
Information Security Resources and Links. Security Certifications, Firewalls, IDS, Microsoft Security, CISSP, Security+

The answer to your problem is to gather the statistics. The bots are usually use the same IPs (no more than several addresses). Restrict the bandwidth eaters but again: be careful.
Use the Control Panel to restrict the addresses.
__________________
The Cyber Teacher
http://www.rtek2000.com
http://www.800-webdesign.com/web-master-links.html -Free Web Master's Resources
_________________
Reply With Quote