Search Engine Optimization Forum
> Questions about robot.txt
View Full Version :
Questions about robot.txt
07-08-2006, 12:26 AM
If you don't have anything private, what is the purpose of having a robot.txt...
Any added value for having one?
07-08-2006, 12:42 AM
They are used to tell web crawlers what files they cannot crawl. Having one does not boost your rankings in the SERPs.
07-08-2006, 02:23 PM
Having one also avoids the search engines getting a 404 error everytime they look for one.