That's an interesting story Ron, as is your Googlebot story...

I currently am blocking bots that claim to be legitimate but really provide me no value, such as nameprotect.com and brandimensions.com which are just trying to find trademark violations--spybots.

I also block turnitin.com which is a service looking for plagiarism.

But I would certainly block AllTheWeb or any other bot that got too hungry. I wonder about The Almaden Crawler which is IBM's super-computer enterprise answer to Google. It downloads my whole site every couple of days, but I allow it thinking that they may someday direct some important visitors my way.

But if I ever have bandwidth issues--which would be at a level about 20 times present traffic--I'd start blocking IBM and other crawlers that don't send me enough identifiable traffic.

Andi

__________________
...the Rockies may tumble, Gibralter may crumble... G & I Gershwin, 1937