Re: Slurp Chewing up 20+ gigs per Month per Site
Yahoo is one of the most voracious bots on the Net. However there are a few things one should do to keep the ravenous appetite of Yahoo in check.
1. Restrict the size of the custom 404 error page. Most of the time Yahoo bombards the site looking for nonexistent pages. If the 404 error page is less than a few Kb it lowers the bandwidth consumed to some extent.
2. If your site serves dynamic pages, try and implement some form of control (like mod_throttle) which will slow down the serving of pages and save your server from coming to its knees.
3. If Yahoo is showing undue interest in your images (reason for massive consumption of bandwidth), deny access to Yahoo in the image directory.
|