As many of you may or may not know, I've relaunched my company site a little more than 2 weeks ago. Having done so, and having added two articles since then to it, here's what I've measured:
Average Google crawler time (from the time I uploaded a change): approx. 27 hours.
Fastest Google crawler time: approx. 21 hours, 45 minutes (discovered earlier today).
Slowest Google crawler time: approx. 31 hours.
Now...MSN and Yahoo!?
Yahoo!/Inktomi has begun to crawl the site over the course of the last week (Wednesday it started, but I'm not sure of the tiem), yet has not finished crawling.
MSN, on the other hand...nothing. The bot has been by repeatedly but for some reason, the old content is still in their index.
Now here's the tricky part...my primary gauge of the crawlers (my site articles) have been indexed on other sites that I've submitted them to after I put them up on my own site, yet not with my domain name (i.e. the articles on adamwebdesign.ca don't show up, but the same ones do on other sites).
The site wouldn't appear to be crawler-unfriendly since Google's pretty well got all of it, and Yahoo! has part of it. But the question becomes: why is Google's so quick to index things permanently (no more Freshbot, it seems) vs. Yahoo!/MSN working much more slowly?