Duplicate Content Question

[noel_x99]

In order to manage source control, I'm working with a programmer to set up development, staging (testing), and production (live) version of our sites. All of these will be on the same server. The development site will always be password protected. Essentially, the staging site is an exact (testing) copy of the production site.

The programmer would rather not password protect the staging area.

My question is will the SE's see it as duplicate content? Is there anything special that can be done to prevent the staging version from being seen as a duplicate?

In theory, there would never be a link to the staging site from any website - so the SE's shouldn't find it. (But I don't want to let it up to chance.)

My first thought was to block all access to the site with a robots.txt - but when that file is moved from staging to production along with all of the other pages - then the SEs would be blocked from the production site too.

We're running these sites on a IIS server.

Any ideas/comments/suggestions are appreciated.

[activeco]

Originally Posted by noel_x99

My first thought was to block all access to the site with a robots.txt - but when that file is moved from staging to production along with all of the other pages - then the SEs would be blocked from the production site too.

Why is that?

'Disallow: /staging' will always remain the same.

[chowell]

The last post is accurate... simply use your robots.txt file to block the spiders from the beta area.

Set a reminder to remove the disallow tag before you go live and/or use separate robots.txt files for the beta & live sites.

You could also use a Meta Robots tag, but you'd have to remove that for launch also.

[noel_x99]

Maybe I'm not explaining it quite correctly. We're using a source control program called TortiseSVN. This program controls the source and as we commit changes, it creates versions of the site.

Suppose we were to put a version - say version 12 - on staging.oursite.com to test it. If everything tested OK, then version 12 would be put into production on the live site: www.oursite.com.

We wouldn't have the opportunity to change a single file - or it would be come version 13.

While search engines shouldn't ever find staging.oursite.com, it will still be there so that we can test version 14, 15, 16...etc before those version of our site go into production.

My concern is that I don't want to have the SEs see that as a duplicate copy of the same site.

I don't think disallow /staging will work because it's not a subfolder. Can I disallow "staging.oursite.com"?

Thanks,
Jane

[subsystems]

What about password protecting HTTP access to the staging site?
IIS lets you to setup a username and password that will require a visitor to login before they can access a site, folder and/or filename.
So just have your hosting company or webserver admin person add the login required info the staging site.

The spiders won't have the login info so they won't be able to spider anything.
The staging site would be for testing and limited access anyway. You can give out the login info only to those that need to test the staging site.

Hope this works for you.
Mike

[SemAdvance]

This should work

Place the robots.txt on the subdomains root

# For domain: http://staging.oursite.com

User-agent: *
Disallow: /

Hope this helps

[craigmn3]

Considering that you can get hosting for next to nothing these days, why don't you simply run your test pages on a seperate site. Block it seven ways from sunday, and when your ready to migrate your test pages to your livesite then download/upload.

this is a less "finese" way of doing it but more secure.

[imvain2]

Originally Posted by SemAdvance

This should work

Place the robots.txt on the subdomains root

# For domain: http://staging.oursite.com

User-agent: *
Disallow: /

Hope this helps

To add to this post, the robots.txt file is looked for per every subdomain. So staging.oursite.com/robots.txt is looked for and www.oursite.com/robots.txt is also looked for. So this solution should work for you.

[tacimala]

Or only allow certain IP's on the staging site.

[Webnauts]

Originally Posted by SemAdvance

This should work

Place the robots.txt on the subdomains root

# For domain: http://staging.oursite.com

User-agent: *
Disallow: /

Hope this helps

I think this would look better. And it will work!

User-agent: Googlebot
Disallow: /

User-agent: *
Disallow: /

[Christiaan]

And then there bots that don't care about the robots.txt file!

[edhan]

Originally Posted by Webnauts

I think this would look better. And it will work!

User-agent: Googlebot
Disallow: /

User-agent: *
Disallow: /

Yes, I agree with Webnauts. This will prevent all bots to scan. I normally will create password protection for all testing sites.

[just-trying-to-help]

Originally Posted by activeco

Why is that?

'Disallow: /staging' will always remain the same.

activeco has got it right.
The spiders will be disallowed in the subfolder/staging and not the index so using the same robots.txt for both the sub folder and the index will be fine.

While search engines shouldn't ever find staging.oursite.com, it will still be there so that we can test version 14, 15, 16...etc before those version of our site go into production.

If it's on the server, chances are Google will find it sooner or later.

Considering that you can get hosting for next to nothing these days, why don't you simply run your test pages on a separate site.

He would still have the same problem of blocking site #2.

SemAdvance wrote:
This should work

Place the robots.txt on the subdomains root

# For domain: http://staging.oursite.com

User-agent: *
Disallow: /

Hope this helps


To add to this post, the robots.txt file is looked for per every subdomain. So staging.oursite.com/robots.txt is looked for and www.oursite.com/robots.txt is also looked for. So this solution should work for you.

Actually, it wouldn't because once transferred to the live site, it would disallow spiders there as well.
Remember: "We wouldn't have the opportunity to change a single file"

Ken

[hoggy]

In your code check Request.ServerVariables to establish the domain\URL in use.

If it's a staging URL, look at client IP address.
If it's not one of your IP's, send a 301 redirect to the live URL.

In the same code you can also change something visual for the staging site (eg add a call to staging.css), so nobody gets them mixed up.

...all code based, so will transfer gracefully from staging to live.

...we've been doing it this way for years.
(Including IIS).

[activeco]

Originally Posted by just-trying-to-help

Actually, it wouldn't because once transferred to the live site, it would disallow spiders there as well.
Remember: "We wouldn't have the opportunity to change a single file"

Correct.

If OP insists on staying on subdomain for staging, one of the solutions could be a server side language such as php or asp to check requested url and provide robots.txt accordingly.
E.g. in PHP something like this:


if (strpos($_SERVER['SERVER_NAME'],"staging"))
{
#provide disallow / version of robots.txt
}

[activeco]

Hoggy,

Either we posted in the same time or I haven't seen your reply. In case of the later, my appologizes.

[noel_x99]

Thanks all! I like serving the robots.txt like hoggy and activeco suggest. I think password protection is an easier idea, but the programmer preferred not to do it. I need to find out why.

[Seminole386]

I have several duplicate sites. The URLs are different and so is the title and a single information pages two clicks away from the index page. Other than that they are identical and all are on the same server. All happen to be linked together with a visit my other sites at the bottom of each page. I rank number 1 on Yahoo, Google, MSN and AOL for all my keywords. Hope that helps.