Robots.txt

[candlese]

Hi- I know what robots.txt is but I have found many answers as to what to do to stop it from being an error - but what is the best way to adress it?? And is it necessary to do so ???
Thanks!

[eightfifteen]

The first question I have is, what error?

[candlese]

The error is in my error logs after a robot traverses the site. I know I can stop them by adding a text file with the User-agent: * Disallow: /- but I am wondering if the error is something I should rectify? Be concerned with?
I am also aware you can use the meta tags to disallow a robot- but this is not fool proof.
With so many robots on the scene it is difficult to list those you want and those you don't unless you simply say - no to all. Wondering what the best way to deal with robots.txt is? Thanks for any help

[eightfifteen]

I think you might be confusing robot.txt and spiders. Robot.txt tells spiders where to go in your site to index, and where not to go.

Here's what I have for my Robots.txt:

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/

What that says to the spider is…Index everything, but bypass the cgi-bin and images directories.

[candlese]

Hi thanks- your right of course- what it should say is after a spider traverses the site I get an error on my log-robot.txt not avail-occasionally! I have used meta tags to dis-allow spiders but it does not work to avoid the error. I also have a robots.txt file- thanks for your example- this is what I have also. I meant to say spider where it says robots- sorry. I guess I will just stay with what I have- and figure its not a problem. Thanks for your help!

[paulhiles]

Originally Posted by candlese

I also have a robots.txt file- thanks for your example- this is what I have also. I meant to say spider where it says robots- sorry. I guess I will just stay with what I have- and figure its not a problem.

If you're using a robots.txt file, it should be placed in the root folder of your site. If a robot attempts to read the file at www.candlesetcetera.com/robots.txt it will produce a 404 error because the file is not present.

Paul