View Single Post
  #7 (permalink)  
Old 07-11-2007, 10:02 PM
shannonlp shannonlp is offline
WebProWorld Member
 
Join Date: Nov 2006
Location: Seattle
Posts: 64
shannonlp RepRank 0
Default Re: Old Pages Being Indexed By Third Party

Just as a thought I write spiders daily for my job. It could be possible that someone has paid to have a custom spider created to scrape the product information from your site.

If this is the case more than likely it would be from many different IP's. This type of bot does not read any robots.txt and can read java/ajax, captcha images, encoded emails, pretty much everything that you think is safe from a bot.


On a good note if it is a professional spider it will create errors for the owner and it will then be modified to go after relevant pages.

Hope this helps
__________________
Web Designer and Custom Spider Creator
eCommerce and shopping cart information

Last edited by shannonlp; 07-11-2007 at 10:03 PM. Reason: spelling oops
Reply With Quote