Re: How to stop Google from crawling secure content/directory?
This is one of those phylosofical questions. Googlebot is nothing more than a normal visitor who, when asked, will refer others to those pages.
If you allow visitors to that part of the site without the need to login, how are you going to prevent them from refering their friends to that part of the site?
If you really want "secure (not-willing-to-share) information" to be available to only those that you choose, you need to password protect it.
robots.txt or other "just for the search engines" kind of ways aren't the way to go because they aren't meant to block "not allowed" visitors.
__________________
FREE SEO ! Really? YES! All you have to do is implement it!
Follow me on Twitter PeterIMC
|