Thread: Robots/Spiders uses very much traffics
View Single Post
  #3 (permalink)  
Old 07-10-2008, 06:32 PM
incrediblehelp's Avatar
incrediblehelp incrediblehelp is offline
WebProWorld 1,000+ Club
WebProWorld MVP
  
Join Date: Jan 2004
Location: Live in Cincy Now
Posts: 7,575
incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4
Default Re: Robots/Spiders uses very much traffics
Slurp has always been a hungry spider. Have you tried changing the crawl delay in your robots.txt?
__________________
KBK Marketing | Search Marketing Blog
Reply With Quote