Googlebot only reading same two pages this month

[steve0]

Every night this month.. at 1 am +/- an hour, Googlebot/2.1 has been requesting the index page and the robots.txt file from a client's new site, and that's all.. just the two files. Though the site is comprised of several thousand pages.

Is this typical behavior?

The links on the site are standard format, nothing sophisticated, so following them should not be a problem.

Should the text on the index page be altered so it appears 'fresher' to invoke more frequent and possibly deeper crawls?

[sudhani]

Btw, which is the website you are refereing to? It always helps if you can post the url of the website u are refering to.

The possible reasons could be:
1. Robots.txt file >> banning search engines.
2. Not enough incoming links >>> this is not a possibility because google has recently completed a deep-crawl. If it finds your home page, it should find your internal pages also. But more links will always help.
3. Robots meta tag on ur home page.
4. brken links / lack of proper navigation structure for your internal pages.
5. use of sessions, etc in a dynamic website.

If you can post the url, fellow members can throw more light on your specific problem.

[sudhani]

Add to the above, if you are having this tag .. just remove it.
<META name="revisit-after" content="7days">

It only tells google to visit your website wait for 7 days for the next visit. it will only delay crawling