MSN Bot eating bandwidth.

[Easywebdev]

I just checked my site stats and MSN's new beta search bot has eaten 43mb of bandwidth. 1425 hits last night. That is only for a small site with a forum, I wouldnt want to see bandwidth reports for larger sites.

[cyanide]

Yeah, I've been seeing msnbot doing some pretty heavy crawling... not sure if that's good or bad though ... lol

ps: I pm'd you

[Easywebdev]

I would wager (hopefully) that this is the first big push, get everything indexed then the next crawls will look for updated content. If sites got hit like that everytime the bot came by there would be some hefty bandwidth charges.

[brian.mark]

MSN's bot actually brought our database server to a crawl last week. We were getting 50 - 100 hits per second from their bot, which in turn used all 4 CPU's to capacity and used all of our RAM.

I guess they just didn't know their own strength?

[bhartzer]

I've actually blocked MSNBot from crawling certain sites because of its use of too much bandwidth. You can block it in the robots.txt file.

1425 pageviews/43mb of bandwidth is about normal for MSNbot. I would suspect that it's not going to crawl that much on that site every day, though, as it tends to crawl less after it's initial crawl(s) of your site.

[brian.mark]

That wasn't an initial crawl. They've listed us as #1 for all of our key phrases since the beta came out. They've hit us like this 4 times, doing 500 or so pageviews per minute for the course of 15 to 20 minutes. This seems to be in addition to the normal crawling that they do to our site. They just seem to go crazy every so often and attempt to kill our database server.

[jrdorkin]

I'm a little upset as well with 15mb sucked up. I'd love to hear some of the bigger numbers as well. Lets see who the BOT got the most out of.....

Anyone have 100+MB??

[brian.mark]

100 MB? Try a lot more than that.

Last month, these are our numbers:

Googlebot: 481.321 MB
Slurp: 995.287 MB
MSNBot: 2136.392 MB

I don't even want to look at the larger of our site's numbers.

Brian.

[Mr Mustard]

The totals for last month (Nov 04) for www.redgoldfish.co.uk were:

Googlebot hits: 51494 bandwidth: 798.19 MB
MSNBot hits: 14878 bandwidth: 392.22 MB

So MSN has a lot to catch up with Google.

[brian.mark]

I talked to a couple of MSN techs at SES in Chicago this week. I asked them about their bot using so much bandwidth and they said it is partially intentional. They want to have the cached version as fresh as possible for the best user experience. They feel that if someone clicks on a page and it doesn't show what their excerpt says it will show, that is a poor user experience. Instead, they're focusing on crawling the pages listed for frequent searches as often as multiple times daily. He said there would be details for telling the MSN Bot how often your content updates posted on their website when their search goes live, but until then they're going to try to figure out how often your content changes through their crawler and start to make adjustments at smarter intervals.

Brian.

[jawn_tech]

I haven't blocked MSN beta yet, but giving it serious consideration. Mainly because my ranking isn't too hot on beta.

Doing well on Google, Yahoo, and current MSN SE. If my keywords were doing as well in MSN beta, I'd let it eat all the bandwidth it's hungry for. If not, I don't see the point...

...that is, until I understand M-beta's algo a little better and have an action plan for doing well, if it stands as-is when they launch.

[jawn_tech]

Scratch that. Looking at my keywords today shows I've bumped up to a #5 spot on beta. Still not as good as the 2 on the current SE, but 5 will do nicely.

[Mac 5]

I hear too many complaints that search engines are eating up bandwidth. I am thankful that the engines are idexing my site and will buy more bandwidth if needed or switch to a different host that allows more bandwidth before I complain. I look at the stats with how many visitors the engines bring instead of bandwidth of robots.