TrafficProducer
04-03-2008, 04:54 AM
Google Adds DIY Robots.txt Creator To Webmaster Tools (http://www.searchnewz.com/latestsearch/senews/sn-4-20080331GoogleAddsDIYRobotstxtcreatortoWebmasterTo ols.html) :p
A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operate on certain data.
A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operate on certain data.