View Single Post
  #2 (permalink)  
Old 01-02-2008, 08:13 AM
thindenim's Avatar
thindenim thindenim is offline
WebProWorld Pro
 
Join Date: Jan 2007
Location: Scotland
Posts: 255
thindenim RepRank 2
Default Re: How to stop Google from crawling secure content/directory?

Add the following to your robots.txt file, where directory is the area you want to disallow. This tells the robot they can't access or index this section.

User-agent: *
Disallow: /directory/*

Alternatively you can use the meta noindex tag on each page: -

<meta name="robots" content="noindex, nofollow">

which will mean that the page is not indexed and links not followed, or: -

<meta name="robots" content="noindex, follow">

which will mean that the page is not indexed, but any links are followed
__________________
Girlz Night - professional hair and beauty products
Web design glasgow - from Thin Denim
Reply With Quote