robots.txt question
what purpose does it serve exactly? my understanding is that you can use it to tell the robots to not crawl certain pages.
if i don't need to tell the bots to stay away from anything on a site, do i need a robots.txt file for any other purpose?
|