|
|
||||||
|
||||||
| Index Link To US Private Messages Archive FAQ RSS | ||||||
| Yahoo! Discussion Forum Yahoo Search discussion. Any topic or subject specific to Yahoo should go here. You will also find a subforum dedicated to YPN & Panama. |
Share Thread: & Tags
|
||||
|
![]() |
|
|
LinkBack | Thread Tools | Display Modes |
|
||||
|
Quote:
I did search of Internet logs and came across this one here. There are numerous entries in this log to view. The crawler resolves to mmcrmX.search.scd.yahoo.com were the X is the number of the crawler. Here are some more sitings of the crawler. It does appear to be requesting graphic formatted files (.jpg, .png, .gif) as well as the directory index of that the files may reside in. If you want to disallow access to images directory, then I would put an exclusion for it in your robots.txt file. To stop them from requesting a directory index of your images -- insert a blank .htm file in your images directory. This will stop them from reading the index and serve up a blank page instead. |
|
||||
|
It appears that the robot is the old AltaVista robot for images which was identified as vscooter in the user-agent field.
It does appear to be operated by Fast though as Elite Skills suggested. Look here. |
|
||||
|
I have noticed the following spiders from one of my client's access logs:
Quote:
Most of those other I don't even recognize. There were some entries for our own WebTrends reporter, but that's the gist of it for July so far. I think that Elite Skills has a good guess for what "Yahoo-MMCrawler" means.
__________________
Cooper Griggs Pure Influence Aloha, surfing and flower stickers http://www.pureinfluence.com/ |
![]() |
|
| Thread Tools | |
| Display Modes | |
|
|
|
WebProWorld |
Advertise |
Contact Us |
About |
Forum Rules |
MVP's |
Archive |
Newsletter Archive |
Top |
WebProNews
WebProWorld is an iEntry, Inc. ® site - © 2009 All Rights Reserved Privacy Policy and Legal iEntry, Inc. 2549 Richmond Rd. Lexington KY, 40509 |