Slurp on a feeding frenzy!

28overpar · #1 (**permalink**) 01-24-2005, 05:11 AM

Can anyone let me know if they have seen any irratic behaviour by Slurp over the weekend?

On one day over the weekend, we had over 20,000 queries on our site, from multiple IP addresses.
At it's busiest it was doing 8 queries a second on our site. (The IP addresses have been verified)

I know we can stem the tide of Slurp with the Crawl-delay command, but I was just wondering what would have caused months of normal behaviour from this bot to get the munchies so heavily and start this feeding frenzy on the site.

Any thoughts?

bbott · #2 (**permalink**) 01-27-2005, 06:34 PM

Yea, the inktomi bot was going nuts on our forum site the past couple days. I think they're updating their index.

greeneagle · #3 (**permalink**) 01-30-2005, 05:05 PM

Seems to me.. That everyone is "Sucking Up" everything they can... Guess the "Full Scale" SE War Is ON!

Ken

Kenai Alaska · #4 (**permalink**) 02-12-2005, 04:23 AM

I just checked one of our forum sites and WOW are all the bots hungry. The campground forum has given up 3.25 gigs to the bots in 11 days. Hope it pays off. And does anyone recognize this IP address 82.80.249.202?
It's going to be Jenny Craig time for Slurp when this is over.

**brian.mark** · #5 (**permalink**) 02-13-2005, 11:14 PM

You were only getting 8 per second? We were getting close to 200 concurrently from Slurp last week. We're looking at some serious database server upgrades because of the traffic they produced. It was slow trying to process a sale during that time, but at least it still worked.

We tried the crawl delay but didn't have any luck with it. They seemed to read it, but ignore it for some reason.

Hopefully we can get some upgrades (pricey, but necessary) before the next time they decide to slurp every page at once again.

Brian.

jltidwell · #6 (**permalink**) 02-13-2005, 11:47 PM

I've put up two new sites in the last couple of months and noticed by my stats that slurp was the first to crawl. I didn't get any from googlebot until I did a small ppc campaign (no skin off my nose, less than $10).

jltidwell · #7 (**permalink**) 02-13-2005, 11:53 PM

Sorry, don't know how that became a duplicate post!

enicola.com · #8 (**permalink**) 02-17-2005, 07:06 PM

I noticed Slurp on my site a lot too. Also noticed googlebot visiting alot more often now that i do rss feeds.

dcallan · #9 (**permalink**) 02-19-2005, 07:18 PM

Yahoo slurp was on my forums on thursday and friday big time, it increased my most users online fig to over 450. The problem is though that it takes so long for Yahoo to actually make the spidered pages live, sometimes about 3-4 months. Let me know how long it takes you guys to get your new pages into Yahoo

montytx · #10 (**permalink**) 02-21-2005, 10:57 PM

I still havent seen slurp very much on my site. Not going in beyond my index.

**brian.mark** · #11 (**permalink**) 02-22-2005, 12:46 AM

Anyone recommend a database server hardware configuration to make our site slurp-proof? It just about stops responding when Slurp comes around now, even with the crawl-delay in our robots.txt file. I don't think they do anything with that directive.

We're using Dual Xeon 2.8's with 4GB of RAM currently. 15,000 RPM drives in RAID 5. Would more CPU or more RAM be a better bet?

Brian.