Re: Old Pages Being Indexed By Third Party
Just as a thought I write spiders daily for my job. It could be possible that someone has paid to have a custom spider created to scrape the product information from your site.
If this is the case more than likely it would be from many different IP's. This type of bot does not read any robots.txt and can read java/ajax, captcha images, encoded emails, pretty much everything that you think is safe from a bot.
On a good note if it is a professional spider it will create errors for the owner and it will then be modified to go after relevant pages.
Hope this helps
Last edited by shannonlp; 07-11-2007 at 10:03 PM.
Reason: spelling oops
|