View Single Post
  #10 (permalink)  
Old 07-05-2007, 11:18 AM
bj's Avatar
bj bj is offline
WebProWorld 1,000+ Club
 
Join Date: Apr 2005
Location: Delaware Valley, PA
Posts: 1,172
bj RepRank 3bj RepRank 3
Default Re: Slurp Chewing up 20+ gigs per Month per Site

Oops, sorry. The mind has trouble believing anyone prefers using a Windoze server in this day and age . . .

I assume you meant keep arachnids out of your image folder?

User-agent: *
Disallow: /

That's the basic format, so if you want to disallow certain spiders from a certain directory, you'd do this:

User-agent: Slurp
Disallow: /images/

More info here:
Yahoo! Help -
Reply With Quote