IMO the issue is much simpler than many of the replies I have read so far on this thread. I gave a run down on my blog here:

Robots.txt >>Search Engine Robots generating too much traffic on your site ?

In a nutshell you can reduce to an absolute minimum robot bandwidth consumption by simply putting an empty robots.txt file in your web or blog.