Re: Search Bots Eating Bandwidth
You can also use this in robots.txt, which should help:
User-agent: * #put spider name in here, or leave it as wildcard
Crawl-delay: 10
And some spiders will recognize this, which is part of the proposed new robots exclusion spec:
User-agent: * #put spider name in here, or leave it as wildcard
Request-rate: 1/5 # maximum rate is one page every 5 seconds
Visit-time: 0600-0845 # only visit between 6:00 AM and 8:45 AM UT (GMT)
Last edited by bj; 06-26-2007 at 03:31 PM.
Reason: clarification
|