Inktomi Slurp or Yahoo Slurp

hunegnaw · #1 (**permalink**) 08-08-2004, 04:35 PM

Several months ago, we discovered that our site was banned from Yahoo! for using a link farm (years ago) but now our ban has been removed and that we're now being visited by InktomiSlurp. They're not long visits - especially compared to GoogleBot which indexes all of our pages. But I again it's InktomiSlurp that show up in our stats tool.

So which is it -- InktomiSlurp or YahooSlurp?

And, how can we get it to index all of our pages?

Thanks,

Dave

jbgilbert · #2 (**permalink**) 08-08-2004, 09:30 PM

Good question!

I do know that this is the official bot name (user agent name of Yahoo's new crawler) Yahoo! slurp

During the early Yahoo transition from Google results, I saw both bots ("Yahoo! slurp" and Slurp") but I have not seen the old "Slurp" bot for quite a while.

ronniethedodger · #3 (**permalink**) 08-10-2004, 06:55 AM

The User Agent field of the Inktomi crawler will have the Slurp identification labled clearly. You will need to view the raw server logs to see this.

Some older log analyzers still feed off of the IP designation which Yahoo has not updated yet (that is according to Tim Mayer of Yahoo). This is the correct way of doing analysis for the User Agent field can be spoofed. The software "should" be updated to properly identify the crawler as Slurp (think of Inktomi as an alias).

If you run the IP of the spider thru a reverse DNS check, you will see Yahoo all over it and the email for abuse, queries, and admin are all Slurp[at]Yahoo.com. In the Answer Section of these crawlers, the DNS information has not been changed -- it will say something like lj1163.inktomisearch.com, but the UA field says Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp).