Thread: Yahoo Slurp
View Single Post
  #2 (permalink)  
Old 10-08-2004, 07:57 PM
tcampione tcampione is offline
WebProWorld Pro
  
Join Date: Jun 2004
Location: Northern NJ
Posts: 284
tcampione RepRank 0
Default
The full database is available at: http://www.robotstxt.org/wc/active/html/index.html

Slurp is #224 (Inktomi Slurp)

user agent is slurp

/*tom*/
Reply With Quote