WebProWorld Part of WebProNews.com
Page One Link To Us Edit Profile Private Messages Archives FAQ RSS Feeds  
 

Go Back   WebProWorld > Search Engines > MSN Search Discussion Forum
Subscribe to the Newsletter FREE!


Register FAQ Members List Calendar Arcade Chatbox Mark Forums Read

MSN Search Discussion Forum Topics and discussions specific to MSN and/or Live Search. There is also a subforum for the discussion of Microsoft AdCenter.

Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 05-30-2005, 10:09 AM
WebProWorld New Member
 

Join Date: Jan 2004
Posts: 18
qoregexp RepRank 0
Default How to slow MSNbot down???

MSNbot appears to be crawling my entire site every day, and not recognizing that because of multiple navigation paths, it's visiting most pages several times. It's using many times the bandwidth of other bots. At MSN's suggestion, I set a crawl delay but it was ignored. Finally in desperation I tried to disallow MSNbot from my site but that's ignored too. Has anyone had any success with getting MSNbot to pay attention to the robots.txt file? Currently I have
User-agent: MSNbot
Disallow: /
as the first thing in my robots.txt file. Have I made some silly error? Thanks for any help.
Reply With Quote
  #2 (permalink)  
Old 05-30-2005, 12:24 PM
brian.mark's Avatar
Administrator
 

Join Date: Jul 2004
Location: Omaha
Posts: 2,717
brian.mark RepRank 2brian.mark RepRank 2
Default Isn't it *?

I thought it was Disallow: *
From my understanding, Disallow: / would only disallow the homepage.

Multiple navigation paths leading to the same page? That shouldn't create a problem, unless you're using different URL's every time. That will also create a problem with Google (duplicate content) and Yahoo for the duplicate reason as well. It'd be a good idea to fix that for all the engines.

We did use the crawl-delay directive for a while, which worked with Slurp immediately but MSNBot seemed to only pay attention after almost a month. Makes me wonder how often they use the Robots.txt rules since I know they do pull it several times per crawl.

Brian.
__________________
ToolBarn.com, an Internet Retailer Top 500 and Inc. 500 Company | Tool Parts | Pet Supplies
Reply With Quote
  #3 (permalink)  
Old 06-01-2005, 09:21 AM
WebProWorld New Member
 

Join Date: Jan 2004
Posts: 18
qoregexp RepRank 0
Default

Might be * ... I have seen examples of both. The problem is the examples are short on explanation of what they achieve.
Reply With Quote
Reply

  WebProWorld > Search Engines > MSN Search Discussion Forum
Tags: , ,



Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Search Engine Optimization by vBSEO 3.2.0