Thread: Robots/Spiders uses very much traffics
View Single Post
  #3 (permalink)  
Old 07-10-2008, 07:32 PM
incrediblehelp's Avatar
incrediblehelp incrediblehelp is offline
WebProWorld 1,000+ Club
WebProWorld MVP
  
Join Date: Jan 2004
Location: Live in Cincy Now
Posts: 7,573
incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4
Default Re: Robots/Spiders uses very much traffics
Slurp has always been a hungry spider. Have you tried changing the crawl delay in your robots.txt?
Reply With Quote