Submit Your Article Forum Rules

Results 1 to 3 of 3

Thread: How to slow MSNbot down???

  1. #1
    Junior Member
    Join Date
    Jan 2004
    Posts
    18

    How to slow MSNbot down???

    MSNbot appears to be crawling my entire site every day, and not recognizing that because of multiple navigation paths, it's visiting most pages several times. It's using many times the bandwidth of other bots. At MSN's suggestion, I set a crawl delay but it was ignored. Finally in desperation I tried to disallow MSNbot from my site but that's ignored too. Has anyone had any success with getting MSNbot to pay attention to the robots.txt file? Currently I have
    User-agent: MSNbot
    Disallow: /
    as the first thing in my robots.txt file. Have I made some silly error? Thanks for any help.

  2. #2
    WebProWorld MVP brian.mark's Avatar
    Join Date
    Jul 2004
    Posts
    2,709

    Isn't it *?

    I thought it was Disallow: *
    From my understanding, Disallow: / would only disallow the homepage.

    Multiple navigation paths leading to the same page? That shouldn't create a problem, unless you're using different URL's every time. That will also create a problem with Google (duplicate content) and Yahoo for the duplicate reason as well. It'd be a good idea to fix that for all the engines.

    We did use the crawl-delay directive for a while, which worked with Slurp immediately but MSNBot seemed to only pay attention after almost a month. Makes me wonder how often they use the Robots.txt rules since I know they do pull it several times per crawl.

    Brian.
    ToolBarn.com, an Internet Retailer Top 500 and Inc. 500 Company | Tool Parts | Pet Supplies

  3. #3
    Junior Member
    Join Date
    Jan 2004
    Posts
    18
    Might be * ... I have seen examples of both. The problem is the examples are short on explanation of what they achieve.

Similar Threads

  1. MSNBot Question
    By sgsmorgan in forum MSN Search Discussion Forum
    Replies: 1
    Last Post: 01-30-2008, 10:04 AM
  2. MSNBot not crawling site
    By Sub01 in forum MSN Search Discussion Forum
    Replies: 4
    Last Post: 04-28-2007, 04:11 PM
  3. Googlebot and MSNbot not visiting
    By jonathanbriggs in forum Google Discussion Forum
    Replies: 19
    Last Post: 11-12-2005, 11:54 AM
  4. wondering MSNbot not crawling in www.fredbelaywatches.com
    By deen in forum MSN Search Discussion Forum
    Replies: 2
    Last Post: 10-08-2004, 10:44 AM
  5. New MSNBot Favoring W3C Validation?
    By greeneagle in forum Search Engine Optimization Forum
    Replies: 8
    Last Post: 02-21-2004, 08:00 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •