Google Sitemap Statistics: Total URLs Indexed

[dmeakings]

Hi everyone.

I added our website (in signature) to my google webster tools account and when viewing the sitemaps overview I found the following:

Sitemap statistics:
Total URLs: 67
Indexed URLs: 38

Would anyone know of any ways and best practices of increasing the amount of URLs actually indexed by google?

I've enhanced quite a bit of the website and created a number of new pages and would like to find a way of getting google to check them out and index them.

Any advice on this is right appreciated.

Thank you.

[wige]

Make sure you have clear and simple navigation to the pages in question. The bots have to be able to find the pages by browsing HTML links, so links contained in Javascript menus and Flash may not get crawled and indexed. Also make sure that the pages in question are not blocked. If you have nofollow on the links or a nofollow meta tag on the page, or if you block a section of your site in robots.txt the pages may not be indexed.

Sometimes, it may just be a matter of waiting. Unless Google decides your content is important enough (total inbound links is a major factor in this), Googlebot may only crawl your pages a few times a week/month. More inbound links to your major pages can increase both the rate that Google crawls those pages and the rate at which Google adds additional pages to their index.

[dmeakings]

Hi wige. Thanks for your comments.

I don't have a robots file let alone one that is blocking anything. None of my links have nofollow attributes, there isn't any flash elements with navigation in them (only one flash object in the header of the main page, very simple), no javascript menus and no blocking meta tags.

Could it be because I don't have a robots.txt file that is giving full permission for the bots to do their work?

[wige]

No. The absence of a robots.txt file should not cause you any problems.

If there is nothing blocking the spiders from crawling to the pages in question, and they are linked from pages that the spiders already crawl, most likely the factor is simply time. It will take a while (again, this seems to depend in large part on the number of inbound links you have and the PR of those linking pages) for the spiders to completely crawl your site. It could be up to a month, although depending on factors ranging from your subject (rarity of the topic) to size of your site, etc. it could be anywhere from a week to a month for Google to do a full crawl.

EDIT: You may also want to check for canonicalization issues - make sure everything is forced to one subdomain (www, or not, but not both).

[dmeakings]

Originally Posted by wige

No. The absence of a robots.txt file should not cause you any problems.

If there is nothing blocking the spiders from crawling to the pages in question, and they are linked from pages that the spiders already crawl, most likely the factor is simply time. It will take a while (again, this seems to depend in large part on the number of inbound links you have and the PR of those linking pages) for the spiders to completely crawl your site. It could be up to a month, although depending on factors ranging from your subject (rarity of the topic) to size of your site, etc. it could be anywhere from a week to a month for Google to do a full crawl.

EDIT: You may also want to check for canonicalization issues - make sure everything is forced to one subdomain (www, or not, but not both).

I have check the canonicalization issues and they have been resolved in my .htaccess file. With respect to time the site has been indexed in google for over 6 months now with many pages missing. There was a time when the title was the same on every page but that has been resolved as well.

I'll have to give it another two weeks I suppose.

[cnxtrans]

I also have the same problem, but my website is one year old.

Sitemap statistics:
Total URLs: 56
Indexed URLs: 36

Some pages were indexed previously and suddenly disappeared from google index, notably the pages linked from the partners page CNX Translation - English / Thai Translations - Partners

However I noticed those pages had all the same META description, so I updated them and will see if things improve.

[Pat]

Hello All,

I'm new to this so please bear with me.

I think I have problem but may not so here goes... any help would be appreciated.

I submitted a Site Map to Google.

In Google Webmaster tools I have seen this:

Filename Status Format Downloaded URLs submitted 10 Indexed URLs 0
sitemap.xml OK Sitemap Oct 26, 2009

as you can see I submitted the Site Map on 26th Oct 09 and it is now 27th could the "Indexed URL's - 0 " be because the GoogleBot has yet to visit my site ? or could there be another problem as the Sitemap.xml file has obviously been interrogated by something because it knows there were 10 URLs in it - I'm a little confused.

[innominds]

Just concentrate on your content and link building activities. Google will automatically find all the urls for itself.

[Pat]

Hi,

Thanks for the quick response.... I will follow your advice.

However I have have tried the google site: command and my site does not appear.

P.S. Any advice / tricks on how to increase links to and from other sites ? as it seems Google likes these