A matter of time...
Google's spider has been running for how long? Since the late 90's.
By comparison, Yahoo! just launched their search engine in Feb / Mar of 2004. Sure, the spider was wandering around before that, but nobody really cared until their search was launched. I think the problem is that it's still evolving, so we don't really know what all it is capable of and people haven't had long enough to make any intelligent observations on how it crawls. If you remember some of the other posts in this area, Yahoo! isn't too good at updating their index, but they sure crawl a lot.
Once we have seen more updated indexes and have had more time to analyze the results and what pages have indexed etc., I'm sure someone will be writing a whole lot about it.
Brian.
|