View Single Post
  #21 (permalink)  
Old 06-27-2007, 07:28 AM
NetProwler's Avatar
NetProwler NetProwler is offline
WebProWorld Member
 
Join Date: Jan 2007
Posts: 88
NetProwler RepRank 2
Default Re: Search Bots Eating Bandwidth

>>38.99.13.123 - - [23/Jun/2007:20:44:45 +0900] "GET /t/imagery/ HTTP/1.0" 404 17507 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)"

That was a 404 for which the server returned something worth 17507 bytes. It might be a custom Page Not Found error page. In a large site where the bandwidth is a worry, avoid using a custom error page with anything more than a couple of KB - if possible. They all add up eventually.

If it is the images which are crawled in excess ( the exact figure will vary depending upon individual circumstances) you can do something to avoid that.

It might be too cumbersome to reproduce it here. But you can see how it is done here:
Targetwoman Blog » Saving Bandwidth in servers

Incidentally, I have not written that blog nor do I have anything to do with that. I just happened to see that.
Reply With Quote