MSNbot appears to be crawling my entire site every day, and not recognizing that because of multiple navigation paths, it's visiting most pages several times. It's using many times the bandwidth of other bots. At MSN's suggestion, I set a crawl delay but it was ignored. Finally in desperation I tried to disallow MSNbot from my site but that's ignored too. Has anyone had any success with getting MSNbot to pay attention to the robots.txt file? Currently I have
User-agent: MSNbot
Disallow: /
as the first thing in my robots.txt file. Have I made some silly error? Thanks for any help.
Submit Your Article
Forum Rules

Reply With Quote
