Massive Google Penalty from crashed server issue

[wesleyw]

I have a site that has been running and building traffic for about 3 years. By December of 2006 it had reached 3 million page views and over 250,000 unique visitors a month.

On Dec 5th Google indexed my site and creashed my server. The DNS agent redirected google to the failover server. Unfortunately the site was in the middle of a massive navagation revamping and the failover server had a mix of old and new source files, including the htaccess.

This broken configuration resulted in creating duplicate content across multiple sub domains. The next day the site was dropped form all the top listings for several very important search terms. Traffic dropped from 25,000 page views to 1000 overnight.

I realized what had happened and disabled the failover server immediately, I also made sure there was no glithces in the production code that would cause duplicate content. Once it was all checked I submitted a reinclusion request to google.

The next day all the cached pages I had in google reverted to september 2006, then about 3 weeks later the cache updated to current and all the old urls were removed.

It has been 6 weeks since I submitted the request and except for the re-indexing, nothing has changed. The home page now only comes up for search terms including the words in the domain name, traffic has stayed horribly low - my alexa rank dropped from 15,000 to 99,000.

The site has had regular growth from 300 visitors a day in January 2005 to 10,000 a day in Dec 2006.
Traffic has inched upward from an average of 1000 per day in december to 1800 a day this month (January) but it isnt even close to the old levels.

Should I submit another reinclusion request? I read somewhere that a duplicate content penalty incurs a minimum 6 month penalty, I read somewhere else that they are more leniant about technical problems.

Any advice would be appreciated, this is costing me an incredible amount of money and stress.

[incrediblehelp]

How many days were you down for in total?

[wesleyw]

It has been down since December 5th - so thats about 54 days

[incrediblehelp]

Sorry to be the bearer of bad news, but that is a long time and their is one reason Google or other search engines shouldn't consider removing websites that go down in this fashion, no matter the backlinks. I can only expect a that a lot of backlinks would help slightly once you go back live. Is the website still down?

[wesleyw]

the site never went down. it crashed during the spider and the failover took over - thats what caused the problem

the main site came back online minutes later and has not been down...

maybe you misread my post

[incrediblehelp]

OK so their wasn't any major downtime? Then you should not have any issues. If the spider comes once and the page is down they WILL come back. They don't come once and see that the site is down and then throw you in the trash. If your failover brought you back online right away you should have been OK. I would look else where for the reason for your ranking drops.

[wesleyw]

When I first responded and said the site was down for 54 days - i meant "down" in the SERPs not that the server went down. (sorry for the confusion there)

Googlebot never stopped spidering my site, it has been spidered daily for the past 2 years. What happened is the site received a penalty 6 weeks ago, on Dec 5th and now hardly comes up for any of its search terms. traffic dropped by 100x

I resubmitted ANOTHER reinclusion request to Google today. Maybe that will help.

[incrediblehelp]

Did you do any design changes or modify your URL or web structure in any way around this time?

[wesleyw]

After the fact. The week folowing the penalty and drop in SERP positions I made changes to the linking structure.

The site is a large directory and before the penalty I had created unique sub-domains for every city-state-category in the directory.

The link structure was:

city_state.category.domain.com

I noticed in the SERPS that the site was doing a little too well, after the 3rd or 4th page for certain search terms, the SERPs were filled with pages and pages of results from our site, all the different subdomains.

So I was working on changing the structure to be state.category.domain.com/city/cityname.html
- to reduce the number of subdomains.