Re: How to stop Google from crawling secure content/directory?
Add the following to your robots.txt file, where directory is the area you want to disallow. This tells the robot they can't access or index this section.
User-agent: *
Disallow: /directory/*
Alternatively you can use the meta noindex tag on each page: -
<meta name="robots" content="noindex, nofollow">
which will mean that the page is not indexed and links not followed, or: -
<meta name="robots" content="noindex, follow">
which will mean that the page is not indexed, but any links are followed
|