View Single Post
  #6 (permalink)  
Old 01-02-2008, 04:51 PM
Peter (IMC)'s Avatar
Peter (IMC) Peter (IMC) is offline
WebProWorld MVP
WebProWorld MVP
 
Join Date: Dec 2003
Posts: 1,485
Peter (IMC) RepRank 4Peter (IMC) RepRank 4Peter (IMC) RepRank 4Peter (IMC) RepRank 4
Default Re: How to stop Google from crawling secure content/directory?

This is one of those phylosofical questions. Googlebot is nothing more than a normal visitor who, when asked, will refer others to those pages.

If you allow visitors to that part of the site without the need to login, how are you going to prevent them from refering their friends to that part of the site?

If you really want "secure (not-willing-to-share) information" to be available to only those that you choose, you need to password protect it.

robots.txt or other "just for the search engines" kind of ways aren't the way to go because they aren't meant to block "not allowed" visitors.
__________________
FREE SEO ! Really? YES! All you have to do is implement it!
Follow me on Twitter PeterIMC
Reply With Quote