In robots.txt, we restrict everything from getting to the cgi-bin.
YET, Google Sitemaps lists over 200 specific pages in the cgi-bin as being restricted by the robots.txt.
The individual pages are NOT listed in robots.txt, just restrict "/cgi-bin/".
Nothing in our site is linked to any of these pages. They're pages over a year old! (We don't use that particular program any more).
It's like Google is reading my directory!
|