Search Engine Spider User Agent

[business-directory]

Hello all,

Does anyone know whether I can trust the top 10 search engines to provide a concistent user agent string identifying themselves?

Reason I am asking is because we want to limit the amount of access to email havesters, and they look a lot like bots, but obviously I don't want to deny access to legit spidering of our site Australian Search Engine and Business Directory, find businesses in Australia clickfind™

By the way, we don't list email addresses, so spidering our site for email addresses is futile, but they try anyway, costing us bandwidth and other resources.

[incrediblehelp]

Does this help?

List of User-Agents (Spiders, Robots, Browser)

[wige]

Bear in mind a bad-bot may have a false user agent. The user agent string is set when the bot is either compiled or run, and can mimic the string of the bot from a major search engine. Most major search engines provide instructions on how to verify that a bot is legitimate (such as these instructions from Google). However, this type of back-tracking could increase your network load.

[business-directory]

Thanks incrediblehelp, thats perfect.

Wige, that link is also excellent, if only ALL major search engines could be verified this way, then we could lock out the rest, i.e. bad ones.

[kgun]

This IP Addresses of Search Engine Spiders is an important site with this Tracking Down Spammers: How to get their contact information classic article with other relevant links.


Gary Keith's Browser Capabilities Project with these downloads Downloads :: Browser Capabilities Project is an exellent resource.