Quote:
Originally Posted by NetProwler
Yahoo is one of the most voracious bots on the Net. However there are a few things one should do to keep the ravenous appetite of Yahoo in check.
1. Restrict the size of the custom 404 error page. Most of the time Yahoo bombards the site looking for nonexistent pages. If the 404 error page is less than a few Kb it lowers the bandwidth consumed to some extent.
2. If your site serves dynamic pages, try and implement some form of control (like mod_throttle) which will slow down the serving of pages and save your server from coming to its knees.
3. If Yahoo is showing undue interest in your images (reason for massive consumption of bandwidth), deny access to Yahoo in the image directory.
|
I also agree that these are some great tips... would anyone know off-hand how I would restrict slurp specifically to my image folder? I think this could help!
Quote:
|
I'm not sure but I believe that Crawl-delay: 10 refers to the timespan "seconds" and not days.
|
Oh I see, that makes sense now.... as for what you said about the .httaccess file, well I'm on a windows server for these sites so it doesn't have one...