View Single Post
  #2 (permalink)  
Old 07-03-2007, 05:46 PM
bj's Avatar
bj bj is offline
WebProWorld 1,000+ Club
 
Join Date: Apr 2005
Location: Delaware Valley, PA
Posts: 1,172
bj RepRank 3bj RepRank 3
Default Re: Slurp Chewing up 20+ gigs per Month per Site

I'm not sure but I believe that Crawl-delay: 10 refers to the timespan "seconds" and not days.

Google's Matt Cutts had this to say about crawl delay, which leads me to suspect I'm right. "I asked the crawl team about this a while ago, and there’s a good reason. It turns out that a lot of webmasters give crawl-delay values that are way out of whack, in the sense that we’d only be able to crawl 15-20 urls from a site in an entire day. I’ll try to post more details about that sometime in the future. The crawl guys are interested in allowing people to give some sort of hostload hint, but it’s their opinion that crawl-delay isn’t the best way to do it."

It's possible that you might be able to do something via .htaccess.

See thread here:
Search Bots Eating Bandwidth
Reply With Quote