In a classic bit of Internet history (1998/12/30), a newsgroup member explained the GoogleBot to a poster complaining of its rude behavior.

"It's a legitimate research project. Unfortunately, it's the worst written piece of crapware you'll ever see crawling your website. It ignores robots.txt (well.... it repeatedly retrieves it, but ignores the contents), ignores robots metatags and headers, gets confused by infinite trees and will suck all your excess bandwidth for significant periods if you let it.

"It's broken. Deny it access.

Cheers,
Steve

(tip from SEORoundTable)

Dan Thies, posting at the SEORoundTable, mentioned a forum thread that told of Microsoft's bot acting in a similarly rude manner.

The forum post, by lundens, states, "the main issues is 175,000 hits and 3 Gig of data just to support msnbot in May thus far is adding up to more than I can afford to use."

"If I were getting hammered like that," said Dan, "I wouldn't rely on robots.txt, I'd block every known MSNbot IP address."

The MSNbot apparently got caught in an infinite loop in a dynamic application part of the poster's website.

__________________
Garrett French
Editor, WebProNews.com
http://www.WebProNews.com

I had to 403 forbid msnbot on May 8th. From May 3rd through 7th it had used more than 4 GBs of traffic. On the morning of the 8th it seemed to be hitting just as hard so I shut it off or else my bandwidth would be all used.

65.54.164.130 - 12930 - 1.36 GB 08 May 2004 - 05:46
65.54.164.87 - 11752 - 1.29 GB 08 May 2004 - 05:51
65.54.164.97 - 11330 - 1.20 GB 08 May 2004 - 06:10
65.54.164.40 - 6799 - 787.97 MB 08 May 2004 - 05:47

namaste zenfort