View Full Version : How to slow MSNbot down???
05-30-2005, 10:09 AM
MSNbot appears to be crawling my entire site every day, and not recognizing that because of multiple navigation paths, it's visiting most pages several times. It's using many times the bandwidth of other bots. At MSN's suggestion, I set a crawl delay but it was ignored. Finally in desperation I tried to disallow MSNbot from my site but that's ignored too. Has anyone had any success with getting MSNbot to pay attention to the robots.txt file? Currently I have
as the first thing in my robots.txt file. Have I made some silly error? Thanks for any help.
05-30-2005, 12:24 PM
I thought it was Disallow: *
From my understanding, Disallow: / would only disallow the homepage.
Multiple navigation paths leading to the same page? That shouldn't create a problem, unless you're using different URL's every time. That will also create a problem with Google (duplicate content) and Yahoo for the duplicate reason as well. It'd be a good idea to fix that for all the engines.
We did use the crawl-delay directive for a while, which worked with Slurp immediately but MSNBot seemed to only pay attention after almost a month. Makes me wonder how often they use the Robots.txt rules since I know they do pull it several times per crawl.
06-01-2005, 09:21 AM
Might be * ... I have seen examples of both. The problem is the examples are short on explanation of what they achieve.