Google not indexing entire site

wturner · #1 (**permalink**) 02-07-2005, 07:10 PM

I have an investment newsletter website that I retooled about 5 months ago. I have waited quite a while for Google to index my entire site. While it looks like it's done all the major pages, it still hasn't done all the one's I'd like.

For instance, it has indexed all the stocks listed on this page: Browse Stock Ratings
However, it hasn't gotten the "B", "C", etc level of stocks - which you would get to if you went through the range listed in the search box.

Also, the old free newsletters located here are not indexed that I can tell: 2003 Archive and 2004 Archive

I think they've got excellent content (at least the newsletters do) and I'd expect them to get picked up.

Is there a reason why the spider hasn't gone for those pages? Is it bored? Too many pages that have similar url's? I don't think having a site index will help as it would have about 5000 links on it.

Of course, Yahoo only has about 51 pages for me, so I guess Google is doing a better job respectively.

Any help would be appreciated.

Also, to go along with the other discussions, the Google update this weekend drastically improved my rankings on lots of keyword. Some from beyond 1000 to top 20.

voasi · #2 (**permalink**) 02-08-2005, 02:22 AM

Get some authority incoming links to internal pages of your site, not just the homepage. This will help get the rest of your deeper pages indexed.

TechEvangelist · #3 (**permalink**) 02-08-2005, 08:55 AM

Google has indexed 248 pages in your site. You can check that using the following query code in a Google search box:

site:turnertrends.com

But it looks like Google is not picking up your title tags, description meta tags or content for most of the pages. At the moment I do not have the time to dissect the pages to look for the problem. If I get some time later, I willl return and take a deeper look.

Nice looking site. Good job with the layout.

wturner · #4 (**permalink**) 02-08-2005, 02:01 PM

I agree with getting the incoming links going. I've been slowly adding some, but most places I submit to never add us. What an irritating process.

I've submitted the home page and the stock-pick-of-the-week page to several places, and have many more to do. I probably need to break down and buy links on some sites.

I also am curious why the description wasn't being displayed on many of the pages Google has listed. In looking, I see that I don't have the keywords tag filled in on those pages. I'm going to add them now, but I don't think that will make a difference. I have different keywords for every page (except the stocks and newsletter pages). However the description is the same for every page. Would that make a difference?

Thanks for looking, and glad you like the layout!

TechEvangelist · #5 (**permalink**) 02-08-2005, 07:55 PM

Both the description meta tag and the HTML title tag should be unique for each page, but that should not stop Google from displaying them. Both should describe the content on the page and the title tag should include your targeted keywords for that page. Google ignores the keywords meta tag, so that won't make any difference.

I've seen this on a few other sites. It may be an anomaly related to the latest update. One thing that you might try is placing the description meta tag right after the title tag, and followed by the other meta tags. Place the link to your style sheet after the meta tags. The author meta tag is worthless, so you can remove that.

I wouldn't bet money that this change will solve the problem, but it is unusual to have the style sheet link before the meta tags.

Jade456 · #6 (**permalink**) 02-08-2005, 08:03 PM

I'm having the same problem. All but a handful of my pages have unique titles, but for some reason not all are being spidered. Although, in all fairness, some are quite new.

Faglork · #7 (**permalink**) 02-09-2005, 08:34 AM

I would start with validating the code. Your homepage has about 40 errors:
http://tinyurl.com/66uap

Incorrect code can cause problems with spiders.

Alex

wturner · #8 (**permalink**) 02-09-2005, 01:44 PM

Thanks for the tips! I have moved the css down below the rest of the meta tags. Also, I've done some more validation on my pages. Most of the errors are from having not properly escaping ampersands. I also didn't have quotes around the 100% width parms.

I don't think either of those are serious enough to throw off spiders, but who knows...

I've got to get a different description going for every page which won't be a small task, but that's probably going to help the most.