View Single Post
  #3 (permalink)  
Old 02-29-2008, 10:11 AM
wige's Avatar
wige wige is online now
Moderator
WebProWorld Moderator
 
Join Date: Jun 2006
Location: United States
Posts: 2,651
wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9
Default Re: Search Engine Spider User Agent

Bear in mind a bad-bot may have a false user agent. The user agent string is set when the bot is either compiled or run, and can mimic the string of the bot from a major search engine. Most major search engines provide instructions on how to verify that a bot is legitimate (such as these instructions from Google). However, this type of back-tracking could increase your network load.
__________________
The best way to learn anything, is to question everything.
Reply With Quote