Indexing of Forbidden Content

[wige]

Another thread (MSN Live Indexes Google Ads) is discussing the appearance of Google Adsense ads in the MSN/Live index. In theory, this should not be possible, as the URLs in question have been blocked through the application of robots.txt. (Line 15 of http://www.google.com/robots.txt specifically disallows the ads.) I can think of a few possible reasons why these pages would be indexed legitimately:

• Google recently changed the URLs but did not update the robots.txt file in time
• For some reason, the Live bot was unable to access the robots.txt file
• There is some form of redirect, where the ads exist at another URL, but the blocked URL is being displayed for the content, possibly an exploit of the Adwords code.

It was mentioned in the thread that it is not uncommon for URLs that are blocked by robots.txt to show up in the indexes of the various search engines. This has severe logistical and security ramifications if true.

Has anyone encountered this issue, where you have blocked content through robots.txt or a robots meta tag and had it indexed? Do you know of any reason why properly blocked content would still be indexed?

[incrediblehelp]

Microsoft Live Search Fixes Problem with Google AdWords Ads

The issue stems from the way Live Search handles content disallowed by the Robots.txt file. We regularly check the robots.txt file of a site to ensure that we don't index and cache pages excluded by the webmaster. However, if we do find a link elsewhere on the web pointing to a page excluded by the robots.txt file, we may include the link and the anchor text in our index if we think it might be valuable to our users. Yesterday we accidently began including the links from the ads of Google AdSense customers. The issue has been fixed, and you should see the results disappear from our search results over the next couple days.

[jtracking]

I just had a thought though...if the page is linked to from another site what happens? does google get to that page before it read robots.txt ?

[Peter (IMC)]

Originally Posted by jtracking

I just had a thought though...if the page is linked to from another site what happens? does google get to that page before it read robots.txt ?

If they find enough links they will index the link,.. but not the content. Generally they use the dmoz title and description in these cases. (if available of course)

[incrediblehelp]

Originally Posted by Peter (IMC)

If they find enough links they will index the link,.. but not the content. Generally they use the dmoz title and description in these cases. (if available of course)

I think you are missing the point here. No matter what they should not index the URL's in this case if they were to adhere to G robots.txt file. Has nothing to do with seeing links fro other websites.

[Peter (IMC)]

Originally Posted by incrediblehelp

I think you are missing the point here. No matter what they should not index the URL's in this case if they were to adhere to G robots.txt file. Has nothing to do with seeing links fro other websites.

I was merely stating what they do, not what they should or should not do.

The discussion is interesting though. Should they index the url or not? It's like being a celebrity. You want your privacy and don't let anybody into your house, but does that forbid magazines to write your name in their articles?

[datetopia]

The perfect crawler should act as a human internet surfer.

It's the same as specifying keywords for the pages. Most engines will index what they consider popular and searcheable information and they will show the results for keywords based on their algorithms and not on website owner's instructions.

[DrTandem1]

Just an FYI, using a robots.txt file(s) does not guarantee either indexing or not indexing. It is simply the designer's preference/request. It may be ignored by certain robots.

[wige]

The issue stems from the way Live Search handles content disallowed by the Robots.txt file. We regularly check the robots.txt file of a site to ensure that we don't index and cache pages excluded by the webmaster. However, if we do find a link elsewhere on the web pointing to a page excluded by the robots.txt file, we may include the link and the anchor text in our index if we think it might be valuable to our users. Yesterday we accidently began including the links from the ads of Google AdSense customers. The issue has been fixed, and you should see the results disappear from our search results over the next couple days.

Expletive. And I mean a really colorful one.

Originally Posted by jtracking

I just had a thought though...if the page is linked to from another site what happens? does google get to that page before it read robots.txt ?

In theory, (and based on what I have read on various webmaster faq type pages) when a search engine "discovers" a url, that url is added to a to-crawl index. When the spider goes through the to-crawl index, it first checks the robots file for the domain, then if permitted it attempts to retrieve the document, and if successful processes and indexes it. At least, this is the way the standard assumes it works, and the way most search engines have claimed they work.

Originally Posted by datetopia

The perfect crawler should act as a human internet surfer.

It's the same as specifying keywords for the pages. Most engines will index what they consider popular and searcheable information and they will show the results for keywords based on their algorithms and not on website owner's instructions.

But there is always certain content that is intended only for humans, not for bots. One example (from the robots.txt web site) is that of a survey. Blocking the script that processes the results should prevent spiders from crawling or indexing the URL, and tainting the survey results.

In the same way, you do not want that URL to be indexed, because the clicks are coming from users who have no idea what they are clicking on, and could taint the results of a survey - or analytics or campaign tracking, etc. In addition, robots.txt is used to keep spiders from indexing login pages for CMS systems and other critical apps to prevent Googlehacking. If I know a certain CMS is vulnerable to attack, and Google indexes a login page for that CMS, I could potentially search for all the sites that use that CMS and attack them.

[TechEvangelist]

I have seen files that are blocked in the robots.txt file AFTER they are already indexed show up in Google's index for almost a year. The issue may be a matter of whether they are indexed before or after they are blocked. I have always found the robots meta tag to be more effective at eliminating URLs from search engine indexes.

MSN seems to be much better at using the robots.txt file to eliminate URLs. Yahoo appears to ignore it most of the time. I've seen blocked URLs show up in Yahoo for years after they were blocked.