Which Spider is this ... ?

kimbecker1 · #1 (**permalink**) 05-12-2004, 07:00 PM

Unknown robot (identified by 'crawl')

Thanks,

Kim

Mel · #2 (**permalink**) 05-12-2004, 10:23 PM

No way of knowing without the log details Kimbecker1, but this is normally a remark which you see in some free stats packages such as AWstats when they do not have the agent name of the spider coded into their program.

One candidate might be the new MSN spider, which few of the free stats packages seem to have included in thier list of agent names.

kimbecker1 · #3 (**permalink**) 05-13-2004, 11:47 AM

I was hoping that maybe it was Yahoo ... I don't see a spider/robot with their name and "crawl" does so daily.

Or is Yahoo Inktomi??

Thanks

steve0 · #4 (**permalink**) 05-13-2004, 01:36 PM

If you have access to your logs.. Find out the IP address the request is coming from..

If you are on a "Unix" type server use nslookup, dig or traceroute to see if you can get more information about it..

Webpages like http://name-space.com/search/ can also be used to do a little reverse lookup..

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode