Hello all,
Does anyone know whether I can trust the top 10 search engines to provide a concistent user agent string identifying themselves?
Reason I am asking is because we want to limit the amount of access to email havesters, and they look a lot like bots, but obviously I don't want to deny access to legit spidering of our site
Australian Search Engine and Business Directory, find businesses in Australia clickfind™
By the way, we don't list email addresses, so spidering our site for email addresses is futile, but they try anyway, costing us bandwidth and other resources.