What is a robots.txt file?

[Tamelyne]

I am designing a brand-new site, which I think the robots have already found, because my webhost's counter info says that I had 7 visitors on Sunday, and there were two errors for requests for the page robots.txt, which I don't even have on my site! Anyway, I don't think I have it... I am using my webhost's template creator thingy, which I hate. In the past, I have written my own html, so I'm not sure what the template creator thing is doing behind the scenes.

So, what is a robots.txt file, do I need it, and what should I include?

The site in question is www.kellys-kollectibles-online.com and it is nowhere near done. I had planned to complete it before I submitted it to the search engines. The pages you can see are what the stupid website creator made. I am working on subpages that will have the actual products being sold, and I will put links to those pages from the website creator templates. (The website creator thing turns your text into images, so when I put Paypal code in for the shopping cart it can't actually make a button!)

Tammy

[cbp]

http://www.webproworld.com/viewtopic.php?t=13110

CBP

[bhartzer]

A robots.txt files is just that, a file named robots.txt that you put in the root directory of your website. There's more information here (http://www.robotstxt.org/wc/robots.html) and a good here (http://www.searchengineworld.com/robots/robots_tutorial.htm).

You really don't need a robots.txt file on your website unless you feel you need to block any spiders from accessing your website. There are reasons why you would want a robots.txt file, though, and here are a few:

1.You have certain files, such as Adobe Acrobat, MS Word, MS Excel, and MS Powerpoint files, which the search engines now include in the search results that you don't want indexed.
2.Certain link-gathering and email-gathering robots are rampant nowadays and telling them to go away via your robots.txt will help reduce unwanted emails.
3.There are programs out there called offline browsers which allow people to download an entire copy of your website. Adding certain entries to your robots.txt file will stop people from downloading your entire site, sometimes for purposes such as copying your web pages for use on their website.
4.You have a lot of images on your website and you want to stop the image search engines from copying all images. If you're a photographer or artist, then you will probably want to stop them from indexing your images.
5.You just want to stop the spiders from getting a 404 error whenever they request your robots.txt file.

[Tamelyne]

Thanks, guys! I read that thread you posted, cbp, and that explained a lot. And bhartzer, those two links you gave are great! I've bookmarked them both, and it looks like the second one has lots of helpful stuff as I work on SEO and increasing the traffic to all three of the sites I webmaster.

Tammy