Robot files

[LottyB]

Hi

I've heard that your site doesn't get indexed properly if you don't have a robots file on the site. Is that correct. If so can someone point me in the right direction for information on creating robot files.

Thanks

[bhartzer]

your site doesn't get indexed properly if you don't have a robots file on the site

That's not true. You really only need a robots.txt file on your site if you wish to exclude the search engine spiders from indexing certain parts of your site.

That said, you still can add a robots.txt file so that certain spiders stay away, such as those spiders that like to find email addresses. The best one I've seen is at WebmasterWorld, see www.webmasterworld.com/robots.txt.

[LottyB]

excellent. That's what I thought. There aren't any files I want to exclude, so would you still recommend doing one?

[markvredeveld]

I am trying to write a robot text file that allows all pages to be followed and indexed by robots but I do not want robots to follow or index any link on any page that starts with the travel now . com -- global discount hotel reservations url or with the travel now . com -- global discount hotel reservations URL. Would the robot text file below accomplish that? Simply stated, I do not want robots to follow links that start with the URLs listed below:


User-agent: *
Disallow: travel now . com -- global discount hotel reservations
Disallow: travel now . com -- global discount hotel reservations