Googlebot gets robots.txt then leaves

[Littlemansearch]

I have been checking my server logs and have noticed that when google bot spiders my website it is now requesting the robots.txt file then that is it,then it will return 30-60 minutes later and request the index page does anyone know why they do this?


Thanks in advance

littlemansearch
Littleman Search Engine Home Page

[Webnauts]

For example you have in your robots.txt:

user-agent: Slurp

Although commands are not case sensitive, I advise you to write exactly "User-agent", that is all lowercase except for the capitalized "U".

Maybe that can help.

[Littlemansearch]

Thanks for the reply I will try changing the U to a capital although I can`t see this making much difference.I had problems with the robots.txt file untill recently which I fixed and everything was working fine when my site was crawled this has only just started happening again I was wondering if maybe it could have something to do with the way google crawls sites maybe they are changing the way they crawl sites.
I know that the search engine I run will only let me crawl every link on a site when it runs on auto,which can be a bit of a pain for other webmasters.I am introducing some new routines to let the engine spider only a few links on a site when on automatic run then move onto the next domain and spider a few their then return to these sites another time.
Perhaps google are changing their crawling technique also.Just a thought.


Thanks

Littleman Search
http://www.littlemansearch/index.html

[Dubbya]

Do you have a Google WebMaster Tools account? Set one up and check their spider summary page for web crawl errors such as broken links, server errors or missing files.

I noticed that you don't have an .xml sitemap. Index your site and create an xml sitemap with GSiteCrawler Google Sitemap Generator for Windows :: GSiteCrawler

You can easily add a link to your new sitemap file in your robots.txt file with the following string:

Sitemap: http://www.littlemansearch.co.uk/sitemap.xml

# end of file

Check your site for broken links with Xenu Link Sleuth
Find broken links on your site with Xenu's Link Sleuth (TM)

[incrediblehelp]

Spidering robots first then index is not abnormal. To be honest if your only seeing spider activity on the home page page you probably need some higher quality IBLs and have them point to inner pages.