 |

11-13-2004, 05:02 AM
|
 |
WebProWorld Veteran
|
|
Join Date: Apr 2004
Location: Donegal, Ireland.
Posts: 322
|
|
MSN Bot eating bandwidth.
I just checked my site stats and MSN's new beta search bot has eaten 43mb of bandwidth. 1425 hits last night. That is only for a small site with a forum, I wouldnt want to see bandwidth reports for larger sites.
__________________
"I have not failed. I have found 10,000 ways that don't work" - Thomas Edison.
"The secret to creativity is knowing how to hide your sources" - Albert Einstein.
|

11-13-2004, 05:38 AM
|
|
WebProWorld 1,000+ Club
|
|
Join Date: Jul 2003
Location: Toronto, Canada
Posts: 2,193
|
|
Yeah, I've been seeing msnbot doing some pretty heavy crawling... not sure if that's good or bad though ... lol
ps: I pm'd you
|

11-13-2004, 06:32 AM
|
 |
WebProWorld Veteran
|
|
Join Date: Apr 2004
Location: Donegal, Ireland.
Posts: 322
|
|
I would wager (hopefully) that this is the first big push, get everything indexed then the next crawls will look for updated content. If sites got hit like that everytime the bot came by there would be some hefty bandwidth charges.
__________________
"I have not failed. I have found 10,000 ways that don't work" - Thomas Edison.
"The secret to creativity is knowing how to hide your sources" - Albert Einstein.
|

11-29-2004, 06:08 PM
|
 |
Administrator
|
|
Join Date: Jul 2004
Location: Omaha
Posts: 2,717
|
|
Darn MSN bot
MSN's bot actually brought our database server to a crawl last week. We were getting 50 - 100 hits per second from their bot, which in turn used all 4 CPU's to capacity and used all of our RAM.
I guess they just didn't know their own strength?
|

11-29-2004, 07:18 PM
|
|
WebProWorld 1,000+ Club
|
|
Join Date: May 2004
Location: Dallas, Texas USA
Posts: 1,569
|
|
I've actually blocked MSNBot from crawling certain sites because of its use of too much bandwidth. You can block it in the robots.txt file.
1425 pageviews/43mb of bandwidth is about normal for MSNbot. I would suspect that it's not going to crawl that much on that site every day, though, as it tends to crawl less after it's initial crawl(s) of your site.
|

12-06-2004, 12:22 AM
|
 |
Administrator
|
|
Join Date: Jul 2004
Location: Omaha
Posts: 2,717
|
|
That wasn't an initial crawl
That wasn't an initial crawl. They've listed us as #1 for all of our key phrases since the beta came out. They've hit us like this 4 times, doing 500 or so pageviews per minute for the course of 15 to 20 minutes. This seems to be in addition to the normal crawling that they do to our site. They just seem to go crazy every so often and attempt to kill our database server.
|

12-09-2004, 10:22 PM
|
|
WebProWorld Member
|
|
Join Date: Oct 2004
Location: Los Angeles
Posts: 92
|
|
MSN bot
I'm a little upset as well with 15mb sucked up. I'd love to hear some of the bigger numbers as well. Lets see who the BOT got the most out of.....
Anyone have 100+MB??
|

12-10-2004, 05:09 PM
|
 |
Administrator
|
|
Join Date: Jul 2004
Location: Omaha
Posts: 2,717
|
|
100 MB?
100 MB? Try a lot more than that.
Last month, these are our numbers:
Googlebot: 481.321 MB
Slurp: 995.287 MB
MSNBot: 2136.392 MB
I don't even want to look at the larger of our site's numbers.
Brian.
|

12-13-2004, 07:22 AM
|
|
WebProWorld New Member
|
|
Join Date: Dec 2004
Location: UK
Posts: 11
|
|
The totals for last month (Nov 04) for www.redgoldfish.co.uk were:
Googlebot hits: 51494 bandwidth: 798.19 MB
MSNBot hits: 14878 bandwidth: 392.22 MB
So MSN has a lot to catch up with Google.
|

12-18-2004, 11:32 PM
|
 |
Administrator
|
|
Join Date: Jul 2004
Location: Omaha
Posts: 2,717
|
|
Talked to MS techs
I talked to a couple of MSN techs at SES in Chicago this week. I asked them about their bot using so much bandwidth and they said it is partially intentional. They want to have the cached version as fresh as possible for the best user experience. They feel that if someone clicks on a page and it doesn't show what their excerpt says it will show, that is a poor user experience. Instead, they're focusing on crawling the pages listed for frequent searches as often as multiple times daily. He said there would be details for telling the MSN Bot how often your content updates posted on their website when their search goes live, but until then they're going to try to figure out how often your content changes through their crawler and start to make adjustments at smarter intervals.
Brian.
|

12-20-2004, 12:06 PM
|
 |
Moderator
|
|
Join Date: Jun 2004
Location: USA
Posts: 1,768
|
|
I haven't blocked MSN beta yet, but giving it serious consideration. Mainly because my ranking isn't too hot on beta.
Doing well on Google, Yahoo, and current MSN SE. If my keywords were doing as well in MSN beta, I'd let it eat all the bandwidth it's hungry for. If not, I don't see the point...
...that is, until I understand M-beta's algo a little better and have an action plan for doing well, if it stands as-is when they launch.
|

12-20-2004, 02:15 PM
|
 |
Moderator
|
|
Join Date: Jun 2004
Location: USA
Posts: 1,768
|
|
Scratch that. Looking at my keywords today shows I've bumped up to a #5 spot on beta. Still not as good as the 2 on the current SE, but 5 will do nicely.
|

12-28-2004, 08:55 AM
|
|
WebProWorld Veteran
|
|
Join Date: Sep 2003
Posts: 328
|
|
I hear too many complaints that search engines are eating up bandwidth. I am thankful that the engines are idexing my site and will buy more bandwidth if needed or switch to a different host that allows more bandwidth before I complain. I look at the stats with how many visitors the engines bring instead of bandwidth of robots.
|
| Thread Tools |
|
|
| Display Modes |
Linear Mode
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is Off
|
|
|
|