Re: Slurp Chewing up 20+ gigs per Month per Site
well nothing has stopped it, I added this to the robot file last weeK'
User-agent: Slurp
Disallow: /graphics/
User-agent: Slurp
Crawl-delay: 5 (was 50)
Slurp has already chewed up 5 gigs on this one website this month alone.
I'm now going to try it at 200
|