How to Prevent Site From Being Indexed...

nashville · #1 (**permalink**) 07-09-2006, 12:43 PM

I have a "mini site" on the same domain with the same contnet as another folder and I want to make sure I do not get a duplicate content penalty.

I added that folder to my robots.txt file with
Disallow: /career

Is that all I need to do? I only use this folder for Google adwords.

Just wanting to make really sure :-)

Thank you!

Webnauts · #2 (**permalink**) 07-09-2006, 02:26 PM

Only for Google?

Then use:

User-agent: Googlebot
Disallow: /career

If you want that for other engines too, then add underneath this too:

User-agent: *
Disallow: /career

bj · #3 (**permalink**) 07-09-2006, 02:29 PM

If you're running sitemap spiders of any sort make sure to filter that directory out of their spidering as well. Some of these programs use robots.txt for filtering but some use their own filter system.

nashville · #4 (**permalink**) 07-09-2006, 02:29 PM

Thanks very much, just wanted to be real sure.