Isn't it *?
I thought it was Disallow: *
From my understanding, Disallow: / would only disallow the homepage.
Multiple navigation paths leading to the same page? That shouldn't create a problem, unless you're using different URL's every time. That will also create a problem with Google (duplicate content) and Yahoo for the duplicate reason as well. It'd be a good idea to fix that for all the engines.
We did use the crawl-delay directive for a while, which worked with Slurp immediately but MSNBot seemed to only pay attention after almost a month. Makes me wonder how often they use the Robots.txt rules since I know they do pull it several times per crawl.
Brian.
|