I have just found out my bandwidth on my one of my windows servers has quadrupled in the last few weeks due to Slurp going crazy... taking around 20 + gigs per month on some of my sites each that have a lot of dynamic product pages. Unfortunately this is not turning into the same increases in traffic & sales and the only increase I'm getting is huge bandwidth bills.
I don't want to shut them out completely as I'm sure some of my buying visitors come from Yahoo but I do have to work out a solution here as 20gigs per site a month is just way to much for a spider and this is also going to be putting excess stress on the server.
I have found that
Yahoo do offer a Robot restriction that I can add to my Robots.txt so I'm going to try this. I have no idea though which number I should use as there is absolutely nothing in their documentation from what I can see that tells you what these numbers mean specifically, just higher is going to mean less visits from slurp.
They give this as an example:
User-agent: Slurp
Crawl-delay: 10
So if my website has 30,000 product pages, what would be a good number to set this? I would like to restrict them to only visit once every 10 days and hopefully they would only use around 1 - 2 gigs a month as opposed to 20.
Any feedback and ideas on what number I should use etc would be helpful.
Unfortunately this host is more expensive than your typical Unix host that offers thousands of gigs per month, they only offer 20 gigs of bandwidth with my package but I'm going to ask them to increase this to be in line with other Windows hosts that offer around 30-40 for my multi domain reseller package.