View Single Post
  #4 (permalink)  
Old 01-08-2008, 01:49 PM
wige's Avatar
wige wige is online now
Moderator
WebProWorld Moderator
 
Join Date: Jun 2006
Location: United States
Posts: 2,651
wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9
Default Re: Google Sitemap Statistics: Total URLs Indexed

No. The absence of a robots.txt file should not cause you any problems.

If there is nothing blocking the spiders from crawling to the pages in question, and they are linked from pages that the spiders already crawl, most likely the factor is simply time. It will take a while (again, this seems to depend in large part on the number of inbound links you have and the PR of those linking pages) for the spiders to completely crawl your site. It could be up to a month, although depending on factors ranging from your subject (rarity of the topic) to size of your site, etc. it could be anywhere from a week to a month for Google to do a full crawl.

EDIT: You may also want to check for canonicalization issues - make sure everything is forced to one subdomain (www, or not, but not both).
__________________
The best way to learn anything, is to question everything.
Reply With Quote