Slurp This!!!

[webreporter]

I have a questionf or anyone who can answer this:

Why is Yahoo Slurp looking for pages in my site that don't exist? I am using Webalizer, and can see all the URL either visited or attempted to be visited by bot or human. These are some of the expamples over the last 12 hours:

Host: 66.196.90.129 /thf281198cg.htm
Http Code: 404 Date: Oct 20 07:22:22 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 66.196.91.78 /project.finance.htm
Http Code: 404 Date: Oct 20 06:49:12 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 66.196.90.243 /freebo/montana.htm
Http Code: 404 Date: Oct 20 06:44:50 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 66.196.90.161 /robots.txt
Http Code: 200 Date: Oct 20 06:44:50 Http Version: HTTP/1.0 Size in Bytes: 1067
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 66.196.90.51 /trave%20l/rhodeisland.htm
Http Code: 404 Date: Oct 20 04:25:25 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 66.196.90.164 /Honor.htm
Http Code: 404 Date: Oct 20 02:39:13 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 66.196.91.55 /humbsf970024.stats/payment.htm
Http Code: 404 Date: Oct 20 02:16:10 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 66.196.91.68 /spec-C/MONTHLY_REPORT/TORCH.htm
Http Code: 404 Date: Oct 20 01:54:32 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 66.196.91.132 /second.htm
Http Code: 404 Date: Oct 20 01:48:01 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 66.196.91.113 /t23gp1pp/rhodeisland/geom.htm
Http Code: 404 Date: Oct 20 00:45:29 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 66.196.91.101 /zukan07daihati/iowa/nlw.htm
Http Code: 404 Date: Oct 20 00:31:38 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 66.196.91.119 /auto/hawaii/clients.htm
Http Code: 200 Date: Oct 19 22:01:56 Http Version: HTTP/1.0 Size in Bytes: 6938
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 66.196.90.125 /mortgage/tennessee/main.htm
Http Code: 304 Date: Oct 19 17:21:30 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 66.196.90.167 /robots.txt
Http Code: 200 Date: Oct 19 17:21:30 Http Version: HTTP/1.0 Size in Bytes: 1067
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Ok, all of the 404's are URL's that don't or ever did exist withing this domain. Why are they looking for pages that don't exist?

Any and all information you can provide is appreciated.

[Maximilian]

Why is Yahoo Slurp looking for pages in my site that don't exist? Any and all information you can provide is appreciated.

Greetings - webreporter!

If these pages DID exist at one time & were previously indexed by Yahoo or simply showed up in an individual search query, this makes sense.

If this is the case, I suggest you either implement a 301 redirect or a customized 401 error program, not to satisfy search bots or spiders, but to not loose traffic due to website link rot.

Hope this helps.

Cheers!
Max

[clasione]

This isn't the first I've heard this from someone lately....

Something's up with this....

I've seen in my logs also....

requests for pages that never existed....

This is very interesting - I haven't got a clue yet on what's going on????

[greeneagle]

Looks like about 3 threads should be combined here! These are top threads at the time of posting. Everyone should read before posting. Many times similar experiences are occuring and someone has already posted. Don't know who was first here!

http://www.webproworld.com/viewtopic.php?t=30444
http://www.webproworld.com/viewtopic.php?t=30439
http://www.webproworld.com/viewtopic.php?t=30406

Someone want to integrate this mess?

Ken

[webreporter]

Why is Yahoo Slurp looking for pages in my site that don't exist? Any and all information you can provide is appreciated.

Greetings - webreporter!

If these pages DID exist at one time & were previously indexed by Yahoo or simply showed up in an individual search query, this makes sense.

If this is the case, I suggest you either implement a 301 redirect or a customized 401 error program, not to satisfy search bots or spiders, but to not loose traffic due to website link rot.

Hope this helps.

Cheers!
Max

Thanks, Max. Those URL's never existed on this domain, nothing even close. I would consider implementing a redirect or an error program, but I don't know how to do any of that. Would one error program work for all those different erroneous URL's?

[Maximilian]

Thanks, Max. Those URL's never existed on this domain, nothing even close. I would consider implementing a redirect or an error program, but I don't know how to do any of that. Would one error program work for all those different erroneous URL's?

Yes, you can find 401 error scripts you can easily customize for whatever platform your website runs on for little to no cost by browsing at:
http://www.hotscripts.com

Once configured, these scripts will pop up for a variety of server errors brought on by browser requests, then redirect the user to whatever page you desire within your website.

Cheers!
Max

[sfowler]

I have seen exactly the same. Yahoo! spent a whole morning slurping nonexistent pages from my site. Nonexistent and they never existed. By the way, my site is hosted by Big Y.

There is no real reason to create an error page for them, it is obviously a technical problem.