Thread: Slurp Chewing up 20+ gigs per Month per Site
View Single Post
  #11 (permalink)  
Old 07-06-2007, 06:18 AM
NetProwler's Avatar
NetProwler NetProwler is offline
WebProWorld Member
  
Join Date: Jan 2007
Posts: 88
NetProwler RepRank 2
Default Re: Slurp Chewing up 20+ gigs per Month per Site
As bj pointed out, you can put up a robots.txt in the document root of your server with these lines :
User-agent: Slurp
Disallow: /images/
Reply With Quote