Odd Traffic from Yahoo Slurp bot

**wige** · #1 (**permalink**) 08-24-2006, 01:51 PM

I have recently redesigned my web site, and have been monitoring search engine crawls to ensure that they are being redirected properly, and have started seeing odd requests from Yahoo's SlurpBot. The request details are as follows:
IP Address: 66.196.91.120
User Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Request: /SlurpConfirm404/randomword/pathtopagethatusedtoexist.html

What has me especially confused is, because of the random word, which is different in each request, as well as the SlurpConfirm404 portion, there is no way that the page they are trying to retrieve could exist, so what is Yahoo looking for?

tmaster · #2 (**permalink**) 08-24-2006, 08:21 PM

They are tring to see what your file not found page looks like. And what kind of headers they get.

Many servers do not send the proper headers for a file not found and this causes the not found pages to be indexed.

Its likely you will see more of this.

edhan · #3 (**permalink**) 08-25-2006, 02:03 AM

How about posting your link here so we can have a look to see what is the problem?

**wige** · #4 (**permalink**) 08-25-2006, 09:28 AM

The full link to my site is http://www.ticketwarehouse.com/index.php. I can see that this is an attempt to access my error page, but I am getting three or more such requests a day, and I am not sure why. My site does seem to give a proper 404 code, but my error page is somewhat dynamic, incorporating page suggestions based on the URL the visitor was trying to access. Might this be what is causing so many requests?

edhan · #5 (**permalink**) 08-25-2006, 11:37 PM

I do not see any fault in your site. But you can remove this meta tag:
<meta name="verify-v1" content="Di/w6hCOKdjz86Aca/+Em3hnT8XxCpQwGDrF2+uOisg=" />

Once Google has verified your sitemap, it is no longer needed. Maybe that is the cause?

**incrediblehelp** · #6 (**permalink**) 08-31-2006, 03:34 PM

Quote:

Originally Posted by edhan

I do not see any fault in your site. But you can remove this meta tag:
<meta name="verify-v1" content="Di/w6hCOKdjz86Aca/+Em3hnT8XxCpQwGDrF2+uOisg=" />

Once Google has verified your sitemap, it is no longer needed. Maybe that is the cause?

Probably not true, Google makes you re-verify every so often, but you can always add it back if they do this.