robots.txt question

[kimber23]

what purpose does it serve exactly? my understanding is that you can use it to tell the robots to not crawl certain pages.
if i don't need to tell the bots to stay away from anything on a site, do i need a robots.txt file for any other purpose?

[andyf]

The main use of robot.txt is to tell different search engine spiders not to crawl or index certain sections or pages of your site.

Also you have choice to specify which robots should not get crawl through your site.
Some robots crawls your site very fast and many times which can affect your site’s performance, so using robot.txt file you can keep away such spiders which hurts you.

[incrediblehelp]

Well every major search engine looks for one when they visit your site. It seems they want to find one. I would add one just to avoid 404 errors, regardless of the other benefits of blocking "bad" spiders and directories.

[kimber23]

so, if i'm not trying to tell any bots not to come, what to i include in the file?

[incrediblehelp]

This example allows all robots to visit all files because the wildcard "*" specifies all robots:

User-agent: *
Disallow:

This example keeps all robots out:

User-agent: *
Disallow: /