Robot txt.

[misterc]

Hi experts, hope you can assist a novice having to learn the hard way. Employed couple of guys to produce my site http://www.furthrows.com and http://www.furthrows.co.uk but we made all the obvious mistakes. I have learnt a lot and made changes to optimize. Google now picked home page up and put me in top 10 but not picking up next level of pages which are optimized even better for the relevant search terms. I see sites using a robot txt line in their html; does this invite the spiders to travel to other parts of the site?
Any help appreciated!

[Artyom]

Well Misterc,

first of all - welcome aboard!:-)

Robots.txt is the txt file which the only porpose is to 'tell' Search Engines what to include and what do not include at their indexes. If you do not have this file on your server Google, Yahoo and MSN WILL SPIDER your site content anyway, but if you do it allows you to restrict some pages or subdirs from being spidering by Google and others.

Read here
http://www.searchengineworld.com/rob...s_tutorial.htm
what you need to include at this file

So the direct answer on your question is - nope, robots.txt can't force Search Engines to include all your site pages, it may help you with prohibitation from spidering some parts of your site only.

Actually I believe there are may exist other reasons why your site other pages haven't been picked up to Google top 10. Here are some to consider:

1. You have more inbound (incoming) links from other sites to your home page then to other pages on your site.

2. Check out your pages titles, bodies, links on those pages (achor text) for keywords you target presence.

3. My experience tells me that Google tends to give high ranking for a site home page when he analyzes this site relevancy to a particular keyword.

4. Check out your internal linkage stracture (links between your site pages). Usually home page get the best scores because it has the greatest inbound links quantity in comparing with other pages on your site.