Subdomain, duplicate content, and sitemaps

[rkstevens]

I have recently changed my site by adding subdomains for some of the different heavy traffic areas for the site, to separate the specific content from everything else. In addition to creating a sitemap and robots.txt file for each subdomain, should I remove all of the links from the sitemap of the main domain and block the subdomain folder in robots.txt, or leave it the way it is. My concern is not to be penalized for duplicate content.

[cass-hacks]

If I understand correctly, which I probably don't, it would seem that what you would want to do would be to redirect from the old URLs on the main site to the new URLs in the subdirectories, no?

If that is the case, you likely will have some duplicate content issues one way or another as Google crawls the old URLs it has been using to find them either 404 or 301 to new URLs but I am not sure there is anyway around that. The hope though is that the transition is short enough to not hurt for too long.

But again, I may have totally misunderstood what you are asking.

[rkstevens]

Here is what I originally had:
mysite.com/folder/files.html
all indexed and included with my mysite.com sitemap.
I have changed the links on the site to recognize the subdomain:
folder.mysite.com/files.html

Currently, I don't have a redirect setup, so the old URL also points to the same page.
So I avoid duplicate content issues, do I need to do the following?
1. remove everything from the mysite.com sitemap that is in mysite.com/folder
2. NoIndex mysite.com/folder/
3. Make a new sitemap for folder.mysite.com

or is there other things that I also need to do, or should do instead?

[cass-hacks]

Originally Posted by rkstevens

Here is what I originally had:
mysite.com/folder/files.html
all indexed and included with my mysite.com sitemap.
I have changed the links on the site to recognize the subdomain:
folder.mysite.com/files.html
Currently, I don't have a redirect setup, so the old URL also points to the same page.
So I avoid duplicate content issues, do I need to do the following?
1. remove everything from the mysite.com sitemap that is in mysite.com/folder

Definitely remove the pages from mysite.com that you are moving or you will continually get 404 errors in your Google Webmaster Tools account or, if you 301 Permanent Redirect them to the new folder subdomain, which is what I think you should do, you will start getting sitemap errors from having too many redirects.

Originally Posted by rkstevens

2. NoIndex mysite.com/folder/

You wouldn't want to do that but instead, use (a) 301 Permanent Redirect(s) from /folder/ to the new subdomain. If you just let those pages 404, disallow or noindex them, and let them eventually be removed, which could take some time, you would not only lose any PageRank they have accrued but also, run into duplicate content problems until they are removed.

Please note though that even if you do 301 them, until each page is crawled at the old URL, Google will likely have the new subdomain URL and the old URL in its database at the same time. But, duplicates should be removed as each 301 is seen. For 404s, it often takes a lot longer for them to actually be removed as it seems Google gives them a chance to automagically reappear at some later point in time.

Originally Posted by rkstevens

3. Make a new sitemap for folder.mysite.com

You will definitely need a new sitemap for the subdomain because each subdomain needs its own sitemap.

Originally Posted by rkstevens

or is there other things that I also need to do, or should do instead?

Here is what I think you should do, incorporating what I've written above.

1. Remove the /folder/ URLs from the mysite.com sitemap.
2. Set up a 301 Permanent Redirect from the old URLs to the new. If your page names haven't changed, you should be able to get by with a single redirect for the entire folder but if you changed some or all of the names, you would need a 301 for each page.
3. Create and submit a new sitemap of the folder.mysite.com URLs.

Does that help?