PDA

View Full Version : How to Prevent Site From Being Indexed...



nashville
07-09-2006, 12:43 PM
I have a "mini site" on the same domain with the same contnet as another folder and I want to make sure I do not get a duplicate content penalty.

I added that folder to my robots.txt file with
Disallow: /career

Is that all I need to do? I only use this folder for Google adwords.

Just wanting to make really sure :-)

Thank you!

Webnauts
07-09-2006, 02:26 PM
Only for Google?

Then use:

User-agent: Googlebot
Disallow: /career

If you want that for other engines too, then add underneath this too:

User-agent: *
Disallow: /career

bj
07-09-2006, 02:29 PM
If you're running sitemap spiders of any sort make sure to filter that directory out of their spidering as well. Some of these programs use robots.txt for filtering but some use their own filter system.

nashville
07-09-2006, 02:29 PM
Thanks very much, just wanted to be real sure.