robots.txt and CGI scripts

[Jurgen]

Fellow Webmasters...

Here is a question I would like to be answered:

If I allow thr robots.txt to spider my "cgi-bin" for my .cgi shopping cart script will it be spidered, and does it make sense for SE's ????

I have no problems with my .html files, but what about .cgi's

Input is very much appreciated.

Thanks,
Jurgen
www.absolutelyfabulousflowers.com

[paulhiles]

Hi Jurgen,

I can't give the authorative answer to your question, but I do know that it's pretty much standard practice to disallow robots from spidering a cgi-bin or Perl directory, etc. There have been instances of certain bots that did not conform to standards (Teleport being one of them) and this caused a few headaches for sites a few years back. My advice is if you're anxious or unsure to allow access to a directory, just add it to your robots.txt file. Easily done!

Disallow: /cgi-bin/

Hope that helps

Paul