Google found my test site - Even with Robots.txt limiting access

[rfrazee]

Ugh!!! Last night, I received an e-mail from Google Alerts, with the list of pages they found for the site lejoslearning.com. It looked normal enough, but it was an old article so I checked it out.

It was a copy of the article, as it appears on my test site. My test site is set up as a subdomain.

Example:
http://testsite.lejoslearning.com (not the actual subdomain but you get the idea.

I just checked Google Webmaster tools robots.txt file and it has the correct folder disallowed, as shown in the example below.

User-agent: *
Disallow: /testsite/

But when I go to Google and type in site: testsite.lejoslearning.com it gives me 180 pages in the index.

What am I doing wrong? Do they treat a subdomain as its own separate domain, which would require its own separate robots.txt file? I am confused. I have a feeling this would cause some serious duplicate content issues.

Please advise your thoughts. Thanks.

[spiderbait]

I believe you need to treat a subdomain as a separate domain. This means you need to have a unique robots.txt file for each subdomain.

[rfrazee]

Thanks. I will work on that option to see if I can get it disallowed, and then try to get things removed from Google, Yahoo, etc.

Ryan

[incrediblehelp]

Try password protecting it. It wont rank for long.

[SemAdvance]

A robots txt file is not always followed.

robots crawl servers and happen to follow links they find on the pages contained there in.

Anything on a server is game to being crawled.....

[deepsand]

Each domain and/or sub-domain for which exclusions are desired requires its own robots.txt file .

See http://www.webproworld.com/search-en...fic-links.html

[martindow]

User-agent: *
Disallow: /testsite/

This robots text would have disallowed
lejoslearning.com/testsite
but not the subdomain as you have set it up

[deepsand]

Correct. As set forth, /testsite/ is a sub-directory relative to the root of the domain within which the robots.txt file exists.

[Clarrie]

Ugh!!! Last night, I received an e-mail from Google Alerts, with the list of pages they found for the site lejoslearning.com. It looked normal enough, but it was an old article so I checked it out.

It was a copy of the article, as it appears on my test site. My test site is set up as a subdomain.

Example:
http://testsite.lejoslearning.com (not the actual subdomain but you get the idea.

I just checked Google Webmaster tools robots.txt file and it has the correct folder disallowed, as shown in the example below.

User-agent: *
Disallow: /testsite/

But when I go to Google and type in site: testsite.lejoslearning.com it gives me 180 pages in the index.

What am I doing wrong? Do they treat a subdomain as its own separate domain, which would require its own separate robots.txt file? I am confused. I have a feeling this would cause some serious duplicate content issues.

Please advise your thoughts. Thanks.

I also had a similar problem with development content showing up on google. Surprised me as the dev area a) was restricted by robots.txt and b) there were no links into it for Googlebot to follow.

Simply password protected the area to keep out prying eyes, but it still puzzled me for a while until I came across this post:
http://www.webproworld.com/search-en...le-spying.html

And realised this was the most likely culprit - looks like if G finds new content through other means than crawling, it doesn't bother to check robots.txt...

So if you are using the google toolbar, this might well be how you content has been picked up

[deepsand]

I also had a similar problem with development content showing up on google. Surprised me as the dev area a) was restricted by robots.txt and b) there were no links into it for Googlebot to follow.

Simply password protected the area to keep out prying eyes, but it still puzzled me for a while until I came across this post:
http://www.webproworld.com/search-en...le-spying.html

And realised this was the most likely culprit - looks like if G finds new content through other means than crawling, it doesn't bother to check robots.txt...

So if you are using the google toolbar, this might well be how you content has been picked up

The days of SEs finding resources on the web by blindly crawling it are long gone. The web is expanding much too quickly for that to now be effective and/or efficient, and there is now a plethora of other means of discovering new resources.

That a specific resource has been newly discovered does not mean that its parent site has been completely crawled; and, it is only when such site crawl occurs that the robots.txt file is discovered.