Google spidering issues

**brian.mark** · #1 (**permalink**) 04-14-2005, 04:50 PM

We were doing well in Google for most of our products for the past several years. Earlier this year, the load on our server got to be too high to leave things as they were. We switched to a static HTML site that generates from a database multiple times per day from a real-time dynamic site. I did 301 redirects on all of the old URLs to the new ones.

After waiting for Googlebot for a while, they know about most of the new URLs. I can do site:www.toolbarn.com and see the list of urls, but most of the pages are simply the URL without any contents or cache.

Does anyone have any suggestions here for getting my content indexed again? I've tried patience, getting more links, and then repeated. Now that we're coming into the busy season, I'm ready to quit repeating and start the begging.

Brian.

(Aren't you happy it isn't a sandbox question?)

AndrewX · #2 (**permalink**) 04-15-2005, 12:05 AM

How long has it been since Googlebot did the deepcrawls to get these pages?

**brian.mark** · #3 (**permalink**) 04-15-2005, 01:08 AM

A lot of them have been a month and a half. Some of them are a few days. It really varies. It seems as if they haven't followed to any of the new links, rather they just saw it was a redirect and stopped right there. The real problem is that as this goes on, we lose the pages that had decent rankings, leaving only this "I know of a URL" trail behind.

Of 343 of our products that they've seen the 301's for, they have not crawled 257. We have well over 5,000 products online, but the rest haven't even been seen yet. The only ones that have been seen are products that other sites link to, but most of those haven't been indexed yet - just the URL list that I'm seeing.

Of the 811 category pages that they've found the new URLs for, 687 haven't been indexed.

They appear in our access logs pretty much every day pulling 100's of category pages, but it just doesn't ever seem to make it to the index.

Do I just need more patience?

Brian.

emils · #4 (**permalink**) 04-16-2005, 05:41 AM

I did this many times in the past. From my experience, you need to wait, especially if your site has a lower pagerank on the index page. I've had such redirects on larger sites like yours that took anywhere from 1 month (sites with PR of 5-6) up to 3-4 months (sites with PR of 3-4).
The fact that your URLs are listed is probably a good sign, G has not updated them yet. I wouldn't worry if i were you, but as i said you definitely need patience on that.