Greetings. This has been bothering me for some time now. I've noticed this in the past, primarily for Googlebots...but now am seeing it from others as well.

It appears that some bots are particularly interested in error logs. I may be 100% off base, this is just the only way I can explain what I've been seeing - if I'm off base, please accept my apologies in advance - I'm open to suggestions :)

Going through my error log I often would see oddball errors followed almost immediately by the same error but from a different IP. I started looking into some of the followup IP addresses and they are all from search engine bots. Here's an example below - last week I made a mistake in entering an image URL - took me three tries to get it right. Below are the error log entries - my IP mostly removed - left enough so you can see it's different from the followup. Notice shortly after I made the mistakes, Googlebot attempted the same URL. I've tried to get Googlebot to follow successful requests the same way, but have no noticed any response - only those in the error log:



I've seen that before, always wondered why, but I've seen it before :) Then today - 1 1/2 days later - I noticed Alexa crawling the same exact URLs:



I cannot figure out any other way that Alexa would know to crawl those exact "not found" URLs other than snooping through my error log.

Anyone know why they are interested in error logs? Is there some other explanation? Thanks for your help :)