The main use of robot.txt is to tell different search engine spiders not to crawl or index certain sections or pages of your site.
Also you have choice to specify which robots should not get crawl through your site.
Some robots crawls your site very fast and many times which can affect your site’s performance, so using robot.txt file you can keep away such spiders which hurts you.
|