Quote:
Originally Posted by deepsand
Would it not be more accurate to say that the GoogleBot was blocked, not because the CVS site was compatible only with certain browsers, but because it tested for the browser name/version in a manner that failed to account for the value(s) associated with the GoogleBot - and, quite likely, other bots as well - and therefore actively redirected the bot to the cited page?
|
OK Deepsand. Maybe that was a bad example. But there are for sure cases that sites cannot be crawled fully or partially due to bad formed code. What are bots anyway?