Robots/Spiders uses very much traffics

[sck4784]

Hi All,

I noticed that Google bots and Yahoo bots uses very much traffics on one of my site.

For example:

In the Month of May:

Googlebot uses 30 GB
Yahoo Lurp uses 11 GB

while in the month of May it was Googlebot 12.04GB Yahoo 6GB.

I don't want to waste my bandwidth unnecessary. Can anybody tell me how I can control on it or what can be done to reduce it?

I think by using robot.txt file we can restrict any search engine on the site and do control increase & decrease crawling rate but apart from this is there any other possible way to control on Google bots & Yahoo.

Please do assist me in this regard.

[textlink969]

i will also like to know about this

[incrediblehelp]

Slurp has always been a hungry spider. Have you tried changing the crawl delay in your robots.txt?

[lspence]

That is a lot of bandwidth.

Do you have a lot of large images? If so you could work on optimizing their file size.

You could also optimize your html. Put all of your formatting into a CSS file. I noticed that you have javascript in every page. You should pull it out of the html, and have it in a separate file. That will make your html files smaller, and faster to load. Good for your visitors, and the bots. Besides, spiders prefer to get to the content of your webpage with out digging through code.