-
04-04-2007 07:08 AM
Permalink: #1
Prevent referrer spam, block bad bots from stealing bandwith
Bad bots and bad sites steal bandwith, courrupt your log files and in the worst case, your site may crash. There are verious methods to prevent referrer spam and block bad bots:
1. Robots exclusion.
http://www.robotstxt.org/wc/exclusion.html
Example:
http://multifinanceit.com/robots.txt
2. Use .htaccess.
Examples:
http://multifinanceit.com/htaccess.txt
http://multifinanceit.com/htaccess1.txt
3. Set a spider trap.
http://techpatterns.com/downloads/spider_blocking.php
http://danzcontrib2.free.fr/en/pieges.php#bloquer
"Setting a Spider-trap
The best method of identifying bad bots is to create what is known as a Spider-trap. Create a directory, block that directory to all agents using robots.txt and link to the directory from a page (usually as a small 1x1 pixel link).
Only bad bots will access that directory (ie they've ignored our robots.txt exclusion). These bots can then be directed to a script that will immediately grab their IP address, User Agent or Referrer and add it to an .htaccess file - so that they're banned from the site".
4. Related links.
Block referrer spam
http://www.ilovejackdaniels.com/apac...referrer-spam/
IP Addresses of Search Engine Spiders
http://www.iplists.com/
SpiderTrack.org
http://www.spidertrack.org/index.php
Prevent hijacking:
http://www.loriswebs.com/hijacking_web_pages.html
link where you can download files.
http://www.garykeith.com/browsers/downloads.asp
Why reinvent the wheel?
Similar Threads
-
By kgun in forum Search Engine Optimization Forum
Replies: 17
Last Post: 06-24-2009, 07:28 PM
-
By vittorio in forum Google Discussion Forum
Replies: 1
Last Post: 05-01-2008, 03:02 PM
-
By Ozzman in forum Google Discussion Forum
Replies: 25
Last Post: 02-22-2008, 02:10 PM
-
By node-net in forum Internet Security Discussion Forum
Replies: 1
Last Post: 05-18-2005, 12:59 AM
-
By tcady in forum Search Engine Optimization Forum
Replies: 3
Last Post: 12-21-2004, 09:59 PM
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
Forum Rules
Subscribe to the Newsletter FREE!
WebProWorld is an
iEntry, Inc. ® site.
© 2010 All Rights Reserved
Privacy,
Legal.
2549 Richmond Rd. Lexington KY, 40509