Quote:
Originally Posted by Webnauts
Can there be any effect somewhere/somehow, if a robots.txt file of a web site has a certain PageRank?
|
If you are referring to an effect on the rest of your site, no. The spiders look at the file purely as text instruction to guide their behavior. The file can't pass pagerank anywhere because it does not contain crawlable links. However, in theory if the page rank were to become high enough, the robots.txt file itself might show up in a search - for a file path for example, or search terms that resemble a file path.
Outside of
SEO, most robots.txt pages are indexed, and you can search for them. Googlehackers sometimes do this to search for hidden areas of web sites or to find vulnerable CMSs. Do a google search for inurl:"robots.txt" and note the top few results.
Quote:
Originally Posted by rumblepup
Strange, because how athoritative can a text document be?
|
Back in the early days of the web, and the early days of Google, a lot of educational resources and even company data files were uploaded as plain text, so the spiders learned to index them, even though they don't contain links, because they can often be the file a user is looking for. Although this is less common now.
Here is an interesting robots.txt file: http://www.webmasterworld.com/robots.txt
All this file contains is commented out blog entries, no actual robots.txt commands. Would this be considered spam?