Subdomain Leaked Linking causing Duplicate Content Filter on SERP?

[arius]

Just realized that due to a sealing mistake when we were creating few of our subdomains, we were in fact creating duplicate content. See examples here:

site:faq.testcountry.com - Google Search

the results that has the ?cat=, ? products.html?, products.html? details.html are all there because there was a linking problem in the menu structure of subdomains that send the java script links to go to database driven pages in fact creating duplicate content like:

Home Drug Test Kits, Urine, Hair, Saliva, HIV, Ovulation, Diabetes - TestCountry

faq.testcountry.com/details.html?cat=121 (NOW Giving 404 since we fixed it but was giving the same content as the first link)

http://faq.testcountry.com/products.html?cat=121 (NOW Giving 404 since we fixed it but was giving the same content as the first link)

There were literally hundreds of cases like this. Also not only in this subdomain but on our other mode/rewrite subdomain that we created here

Mold Tests - Health Hazardst - Mold Testing Kit
vs..
testingkits.testcountry.com/products.html?product=1211

The sole reason we wanted to do a sitemap based mode/rewrite was the assumption that robots can not read javascript and our dynamic database driven product and category pages. Now we see that we were not correct.

This duplicate content issue is causing declines in SERPs, that is pretty obvious since the site receives almost the same organic traffic from MSN or ASK as Google.

What to do next, you think?

-we are going to seal all our subdomain links, and then give 404 errors to duplicate subdomain pages
-Ask for reinclusion with webmastertools on Google (Do we need to since w are already included it is just we are penalied for duplicate content)

[arius]

Anybody else have the same subdomain based duplicate content problem ?

[incrediblehelp]

I dont think you need to ask for reinclusion just eliminate the duplicates from the website and then ask for removal of the duplicates through Google Webmaster Tools.

[raja4]

I had the same problem on one domain. We sealed the site almost 6 months ago but have not seen any improvement so far.

[arius]

Raja, thanks for sharing information. I did not expect to just fix the errors and hoped that all will be good very quickly without an automatic update. I have been around long enough to know that it aint that easy.

It is just frasturating that as webmasters and web site owners we do not have means to fix things like this rather quick. I do believe

- Search Engines, just like Credit Score companies or Phone Companies have responsibility for the end users (visitors) and website owners in this space.

- Just like Government regulates credit score companies or phone companies to operate with fairness , openness, with means to change/fix mistakes -within reasonable amount of time- , I think she should regulate Search Engine companies.

- As website owners we should have the right to be able to get a response from Google & Other Search Engines if we were to be penalized / filtered / sandboxed etc... Search Engine being how crucial they are for our new World (Google being the #1 Master of the universe) should have that responsibility. AFterall Google first, Search Engine enjoy the benefits of high market capitalization, high revenue and more in this new space.

Anyway I guess that is a long philopshical subject. To me now important thing is:

Problem is fixed, how to get Google to not penalize me now. (Other than waiting for the God to see I am a good boy that fixed a technical problem)

[saxman1973]

If I am understanding you correctly, you are doing mod re-writes to help with your SERPs because your site uses javascript. Then you state that "Now we see that we were not correct.".

You were correct. Spiders cannot follow javascript links. You can however remedy this with use of CSS navigation, but it looks as if you are using a product cart of some sort though. There are product carts that use CSS navigation, but they still use the dynamic URLs. The best way around this is to do as you suggested and implement URL re-writes. It will take some time if you have alot of pages, but is well worth it in the end.

As far as the duplicate content is concerned "incrediblehelp" is right on...

"I dont think you need to ask for reinclusion just eliminate the duplicates from the website and then ask for removal of the duplicates through Google Webmaster Tools."

Hope I understood you correctly. If so I hope this helps some. - Saxman

[saxman1973]

Early Impact has a great product cart that has a CSS navigation that is fully customizable. - Saxman

[wige]

As far as trying to get the search engines to respond to your removal of the duplicated links, beyond what you are already doing, I would suggest that you consider changing from responding to the requests for the duplicated pages with 404 messages, and instead use 301 redirects to point to the original content. There are a few reasons that this may be preferable:

1) Increased speed. Search engines may leave 404 pages in the index for up to six months. This is correct behavior according to the specification. A 404 error message means that a resource is currently unavailable, generally due to a temporary issue - the file was removed for maintenance, a background server is down, etc. A 301 redirect on the other hand means that the requested file is gone, will never return, and should not be requested again.

2) Matches search engine behavior. Search engines try to be as adaptive as possible to keep up with changes to the web. Part of that adaptation is spotting redirects and new file locations and "merging" records in the index accordingly. By redirecting the duplicated pages to the original page, you are helping the search engines spot that the content was duplicated and the problem has been corrected in an explicit manner. The index may need to go through a discovery process to discover the change otherwise - the index has detected that three different pages are identical, and with 404s you have to wait for the index to spot one page is removed, and update the original page accordingly. With a 301, you are explicitly showing the search engine the link between the two URLs merging the records and forcing an update of the data for the surviving version which may remove black marks more quickly.

3) Increased crawl rate. As the search engine sees the redirects, although it may not recrawl the target page immediately, it will schedule the target to be crawled as soon as possible to check for spam and make sure the redirect is valid. This will again cause the index entry for the target page to be updated, possibly multiple times.

4) Maintaining link and bookmark value. As it seems these duplicated URLs have been in existence for some time, it is possible they have generated inbound links and that users have bookmarked them. By redirecting the user to the non-duplicated pages, you keep any pagerank from the links flowing, and prevent a negative user experience by showing the user the content you know they are looking for instead of an error message.

I am not by any means saying that you must change to do it this way, just that it is something to consider. Others may have other experiences, and my comments are based more on my own experiences as well as various tidbits of SE behavior I have seen posted by search engine sources. Other members may be able to point you in the right direction as far as whether or not this would be a good idea in your situation, but I suspect it might help.

[davidweb]

Duplicate Content Filter is extremely aggressive when it comes to hunting websites with duplicate text.

ONE PIECE OF ADVICE : AVOID DUPLICATE CONTENT AT EVERY COST