View Single Post
  #11 (permalink)  
Old 11-30-2006, 07:16 PM
blitzen blitzen is offline
WebProWorld Pro
 

Join Date: Apr 2006
Location: Earth
Posts: 241
blitzen RepRank 0
Default

In robots.txt, we restrict everything from getting to the cgi-bin.

YET, Google Sitemaps lists over 200 specific pages in the cgi-bin as being restricted by the robots.txt.

The individual pages are NOT listed in robots.txt, just restrict "/cgi-bin/".

Nothing in our site is linked to any of these pages. They're pages over a year old! (We don't use that particular program any more).

It's like Google is reading my directory!
Reply With Quote