YAHOO at Mach5 - Never seen anything like it! [Archive]

greeneagle

05-27-2004, 02:59 PM

Recently we built a new client site. All we did was link and blurb on our portfolio page. Inktomi flew over and slurped it up at Mach5 at least - Before we could manage the final touches. GOOGLE is still dragging behind.

I am standing here in the vapour trail wondering just what happened in awe. What is going on at YAHOO? Has any one else experienced this new speed?

This has never happened before - I am talking some new alien spacecraft or something, never seen anything like it. Immediate spyder and index - Ohhh! - fast!

We should probably expect indexing speed to be a primary competitive issue as they rev up, but I am talking lightening here!

Ken

bhartzer

06-18-2004, 04:49 PM

I'm curious--did the spider come and spider the page and now the page is in the index? Or did it just come spider the page?

I'm seeing the same fast spidering as you, but it usually takes a long time before the page or site ever appears in the index.

janeth

06-18-2004, 05:57 PM

Hi Ken, ignore the text because the site is only hours old and if you think about some old post here on the forum I think you will understand the reason for the site, but we just put www.geeks-webhosting.com online and Google hit it first thing.

I have not seen Yahoo yet but again the site is not even a day old.

Dave Hawley

06-18-2004, 10:13 PM

paulhiles

06-20-2004, 10:47 AM

Just echoing Bill's comments.. I've seen a fair bit of Slurp spidering my sites, certainly an increase on say, 6 months ago.. but the results are unfortunately not reflected in Yahoo's index. I see results there from weeks, if not months ago.. whereas Google returns the same pages updated to within 24-36 hours.

matauri

06-21-2004, 05:19 AM

I see a definate difference lately. I think Yahoo has gained a few more horses. I've noticed it with index positioning and backlinks after a very short period.

Gary Golden

06-21-2004, 04:05 PM

YAHOO is always a day late and a dollar short with me, I usually change my site a couple of time and see the outdated info on YAHOO. Sure hope things get better though!

greeneagle

06-21-2004, 06:11 PM

I am seeing the spydering almost immediately and indexing 3 weeks before GOOGLE on new Sites I blurb on my portfolio page. I will initiate a test on a revised, SEO'd and validated site within the next 2 days.

Ken

smakyyy

06-26-2004, 10:22 PM

somehow yahoo is going faster that i have noticed- also it gives better ranking then google - :-)

cbp

06-26-2004, 10:27 PM

also it gives better ranking then google - :-)

Better for who? Your site?

If you mean better results than Google - the general www searching public do not agree as Yahoo is loosing marketshare to Google

CBP

smakyyy

07-01-2004, 05:53 AM

i am # 376 on google for the term - diamond ring - but i am #2 on google withe the same word. interesting though that after yahoo went away from google my site has been on yahoo as #2 for that search term.
But yes- i want to be on the first page with google - but I like being on yahoo also - btw. i am #8 on MSN and #3 on altavista for the same keywords.

jestep

07-01-2004, 10:23 AM

Yahoo seems to be indexing a site almost as quick as google, but they never update their cache. It is constantly about 3 - 4 weeks behind. Yahoo still has a lot of improvements before it is anywhere close to google in my book.

greeneagle

07-01-2004, 08:57 PM

Is there an explanation for all these differing results here? Does it fall out geographically?

Ken

Dave Hawley

07-01-2004, 09:58 PM

Is it me, or is Yahoo still struggling to index Framed sites?

greeneagle

07-01-2004, 11:03 PM

Dave:
"Is it me, or is Yahoo still struggling to index Framed sites?"

I thought everyone was - still!
LOL
Ken

Dave Hawley

07-01-2004, 11:53 PM

Good lod googlebot seems to have no problems. I know they suggest no using them, but I have Frames and googlebot goes beserk all over my site. In fact, last month it chewed up a whopping 32GB of bandwidth!

greeneagle

07-02-2004, 12:39 AM

It was just trying to find something, anything Dave! What are you doing using frames? Did you at least have your Frames DTD correct?

Ken

Dave Hawley

07-02-2004, 01:18 AM

What are you doing using frames?

It's a long story that boils down to ingorance. I'm slowly in the process of dumping them all-together after 3 years. I guess while Yahoo was using Google they was no real incentive to dump them, now that Yahoo is on it's own (and may join with MSN) it might be time.

Having said this, it would much nicer if Google would show all the other SE how to index Framed sites :)

ronniethedodger

07-03-2004, 12:08 AM

It's a long story that boils down to ingorance. I'm slowly in the process of dumping them all-together after 3 years. I guess while Yahoo was using Google they was no real incentive to dump them, now that Yahoo is on it's own (and may join with MSN) it might be time.

For the record, Yahoo is still using Google results. They will fall back on them in the absence of their own. Google is (per their S1 filing) still under contract to fulfill that obligation until the end of July.

I have noticed a number of people mentioning that it takes Yahoo a few weeks, give or take a day, before they see the updated results in the Yahoo index. Do not confuse this with your site actually being spidered by the Slurp crawler ... it is more like a delay in Google releasing that to Yahoo. Google may be under contractual obligation to feed them results, but that does not mean they have to be fresh results.

Dave Hawley

07-03-2004, 12:19 AM

They will fall back on them in the absence of their own
Is that likley to happen? Can you give an example?

ronniethedodger

07-03-2004, 01:03 AM

They will fall back on them in the absence of their own
Is that likley to happen? Can you give an example?

Yes. No. They go and come, sometimes you will see it.

Example is when Google reverted back to some older cached pages of my website. I do not know why or how it happened, but all of the newer pages they indexed and the updates were gone. It seemed to me at the time that I was looking at the same results I was looking at a couple of weeks prior.

I looked over at Yahoo and everything appeared to be okay over there...had the newer pages and updates, no change. A couple of hours later when I had checked in again, lo and behold they were reflecting the older cached pages and none of the newer ones. It was a spitting image of what happened at Google.

You will hear people refer to Yahoo doing this from time to time across a number of forums. They will say they are reflecting Google results.

Dave Hawley

07-03-2004, 02:02 AM

Ron, there are 2 things here.

1) If it's likley to happen (as you have said) that would strongly suggest that Yahoo are likley to not be able to supply their own SERP's at times. I cannot for the life of me see why this would ever be the case? I have never seen it and I check at least 10 different terms on yahoo each day.

2) If Yahoo were still using Google at times, they would have to dislay the text "Results supplied from Google" as they did when Yahoo were using Google.

It was a spitting image of what happened at Google
There is a difference between a "spitting image" and Yahoo still using Googles results. There are quite a few search terms on Yahoo (so I have read and even seen) that would be the "spitting image" of the Google SERPs (at least for page 1). This is just down to these pages fitting the 'bill' for both Google's algo and Yahoo's. I'm sure this happens across many SE at times.

While I don't dispute that Google are "still under contract to fulfill that obligation until the end of July." I cannot see Yahoo ever needing or wanting this to happen now there have their own database, SE and algo.

ronniethedodger

07-03-2004, 02:52 AM

Some of the references to this at WMW and a clue from Tim Mayer (http://www.webmasterworld.com/forum35/1816.htm) suggests that Google results will appear when they are working on their own internal programming. It is like one switching out, and the Google results are fed off of. This is the only plausible reasoning I can see behind it.

As far as having to show the "powered by Google", I would not think so. But I am not a coorporate lawyer either.

All I know is people see it from time to time, and a thread gets started so it is an intermittant phenomenom. Here is one such thread (http://www.seo-guy.com/forum/printthread.php?t=647), but this one has a twist to it and involves the IE browser only being able to detect it (at this point in time).

I can probably dig up more sightings of this, there may be a couple here at WPW. All I know is I have seen it happen with the back pedaling on my own site and that was evident enough to show they were pulling from Google at that time.

Dave Hawley

07-03-2004, 03:27 AM

Thanks Ron, that's interesting stuff.

ronniethedodger

07-03-2004, 04:18 AM

No problem Dave.

Now the interesting to watch for, and I have been waiting ... is August! It will be an interesting month all around I think. Not only will Yahoo be standing totally on their own two feet without a cane, but MSN will be either rolling out their SE solution or getting very close to it. All of this will be overshadowed (as usual) by one of Google's updates that seems to get named like hurricanes are named and are just as stormy.

This year I think will be looked back on by many people as a major turning point in SE history.

Dave Hawley

07-03-2004, 05:32 AM

Agree, August will be an interesting month. I'm wondering what (if anything) Google have up their sleeve, just in case Yahoo or MSN get it right.

I couldn't stop laughing when ALL the talk was about MSN and Yahoo upping the anti in the SE wars. Then, on April 1, Google, out of the blue, announced gmail :)

ronniethedodger

07-03-2004, 07:55 AM

August, getting near the quarter end too. Some things will asuredly be shaking up. MSN and Yahoo are kind of uneasy bedfellows right now. MSN is pulling results from the Yahoo core. They both have a semi-competitive stake in the partnership in this anti-spam campaign going on. It is like an alliance, that isn't an alliance in some respects.

MSN rolls out with the new layout which clearly identifies paid content. This also meant the dropping of all PFI/SiteMatch from the main results. Yahoo now is the lone PFI bad guy now because of that.

Google is search. Period. In order for them to compete with these two breathing down their neck, what do they do. Gmail for one, but they also are taking search to the desktop eventually and that is the hallow ground of MicroSoft...they have two years to amble on in, get entrenched, then wait for Longhorn to roll out.

The funny thing about Gmail was all the knee-jerk reaction it got from the other two. That was the funniest part to watch. But there is another thing that it did that is very subtle if you look at it closely -- it grew their userbase. Think about it.

Google took the users of blogger and gave them a gift in the form of invites. A precious three each. In one fell swoop they quadrupled that userbase overnight. These are bloggers too handing them out, and they are loyal users at heart and touted Gmail. People were dying to get their hands on a Gmail account. Except there are a few Yahoo nay-sayers (http://www.webproworld.com/viewtopic.php?t=22710) out there now who think that Gmail is a flash in the pan. Right. Read that thread Dave and you will see the real purpose behind Gmail, good read right at the end. ;0)

greeneagle

07-03-2004, 12:49 PM

ROn,
I haven't had the opportunity to really check out GMail. You seem to think it has some good benefits. I don't remember seeing another forum here that explores it's features in detail and how they can be used. If you were to start a thread on that topic or already have one going would you please notify us.

Thanks,
Ken

ronniethedodger

07-03-2004, 05:17 PM

I think a majority of everyone here has a Gmail account Ken. Then there are some who do and don't like it. The ones that don't will often speak up and ask for an invite, of which I have handed out two of them already here. (hint: I have a hidden invitation on my site too ;0)

The point being is that we already pretty much know what the features are. There is however a small debate going on in this thread (http://www.webproworld.com/viewtopic.php?t=22710) between the differences in Gmail and Yahoo mail. Although I think that thread was started by another privacy advocate that seem to be coming out of the woodwork a lot lately now that Gmail is spreading like wildfire.

I take it Ken, that you have an account but just don't use it then? You could drop a few comments in the other thread if you wish.

greeneagle

07-03-2004, 05:27 PM

Actually Ron, I don't have one. I am being completely sincere. Have been so busy that I haven't even looked into it.

Ken

sfowler

07-06-2004, 02:24 PM

I've not seen much of this Yahoo at Mach 5 business. Slurp has not been near my index page yet, let alone anywhere else. Googlebot ate the lot pretty quick, but no sign of Slurp.

greeneagle

07-06-2004, 05:12 PM

It seems like the GOOGLEBot has had a hydraulic breakdown this month so far too. I don't recall a month where I saw 6 or 7 others ahead of it in crawl frequency/depth.

I guess they are redefining the "sinusoidal function", better get ready to ride the waves. Looks like they are getting deeper and decreasing in frequency. That is predictable though as alg adjustments become increasingly complex.

DOn't mean to seem too sarcastic here, but I am sure many view it as a "sawtooth" function!

Ken

sfowler

07-07-2004, 01:13 AM

So is Slurp in the workshop getting a general overhaul right now? I have no objection to the algorithms getting more specific. Searches are also becoming more specific, and I am personally fed up of wading through pages of bovine feces when I am looking for something.

ronniethedodger

07-07-2004, 03:59 AM

So is Slurp in the workshop getting a general overhaul right now?

Has it ever been out of the shop? Outside of a couple of people who have had the privelege of Slurp's presence, it is somewhat non-existant. Although, I was graced by its presence tonight -- twice!! It grabbed a couple of useless pages, so it might as well go back to the shop.

Quite frankly, I do not know where Yahoo is getting its pages for the Index from. Things just are not adding up given the rarity of Slurp and their claim that they have over 2 billion pages. MSNbot has been hell on wheels for about as long and they only claim 1 billion.

Dave Hawley

07-07-2004, 04:12 AM

Looking at my logs it's Googlebot all the way so far this month and every month!

There just seems to be no threat to Google anywhere in sight!

Googlebot (Google) 278607 7.02 GB 07 Jul 2004 - 04:55

Jeeves 1456 31.70 MB 07 Jul 2004 - 01:02

Alexa (IA Archiver) 1005 8.03 MB 06 Jul 2004 - 18:34

Inktomi Slurp 974 5.55 MB 07 Jul 2004 - 04:35

I bet Googles database hits 5 billion before too long. In fact it might be there (or bigger) now, They might be waiting for the right time to make the announcement.

I can see it now. Yahoo will brag 2-4 billion, MSN about the same, then Google will blow them both out of the water and have something like 6 billion!

sfowler

07-07-2004, 10:02 AM

So why do we worry about what Yahoo! wants if they are not indexing changes? There are plenty of links on the site, not counting the forum signatures, so they have no real reason to not at least visit in 6 weeks. Just a visit, they don't have to index everything at once!

greeneagle

07-07-2004, 11:56 AM

Dave:
"Looking at my logs it's Googlebot all the way so far this month and every month!"
_____

I am just not seeing that this month Dave! - GOOGLE is # 6 out of a field of around 15 right now!

The slowest I have ever seen.

Granted it does get stimulated on page revision, How many pages have you revised since last time?

Ken

sfowler

07-07-2004, 04:55 PM

I revise or add about ten per week, which may be what keeps the Googlebot interested. Still absolutely no sign of Slurp or MSN in 6 weeks!

Dave Hawley

07-07-2004, 11:08 PM

Granted it does get stimulated on page revision, How many pages have you revised since last time?

This might be where the difference comes in. I add about 5 pages manually per week. However, I have a popular Q&A forum: http://www.ozgrid.com/forum/ that dynamically creates about 20+ pages per day.

Googlebot seems to thrive on new content and is about the only SE that can truely deep crawl dynamically created pages with few problems.

ronniethedodger

07-08-2004, 03:23 AM

Googlebot seems to thrive on new content and is about the only SE that can truely deep crawl dynamically created pages with few problems.

MSNbot is crawling thru that crap pretty effectively also. It just got done crawling 53 pages in a 24 hour period on our forum.

Googlebot and MSNbot are both going at it like nobodies business. Very aggressive, and they do not seem to get lost on those post, quote, or login buttons all too often like the others do. Actually I should probably put those in the robots.txt file, now that I think of it ... hehehe. doh.

On a good note, Slurp has actually been in yesterday. Actually crawled 3 pages ... wow, it must of been bored or something. ;0)

Dave Hawley

07-08-2004, 03:50 AM

MSNbot is crawling thru that crap pretty effectively also. It just got done crawling 53 pages in a 24 hour period on our forum.

Why do you call it "crap"? MSNbot might be doing this on your forum, but not on mine :(

53 pages in a 24 hour period on our forum
Googlebot gets well over 5000 pages from my forum within 24 hours.

greeneagle

07-08-2004, 02:36 PM

If I had 5000 pages, I'd be retired and enjoying the good life! What's up with that?

Ken

ronniethedodger

07-08-2004, 04:55 PM

Why do you call it "crap"? MSNbot might be doing this on your forum, but not on mine :(

I meant crap as in redundant and long Urls. It was more laziness on my end to do some basic things to shorten them. I was being...ummm...self-critical. They are crappy Urls.

53 pages in a 24 hour period on our forum
Googlebot gets well over 5000 pages from my forum within 24 hours.

Yep, Googlebot can go on a frenzy like that. They will come in with more than one crawler and feed and feed and feed and keep going at it for extended periods of time. I have seen MSNbot do this to some extent too, but not at the rate Googlebot can do it.

If I had 5000 pages, I'd be retired and enjoying the good life! What's up with that?

It is not that hard to do Ken. We have about 2000+ pages. Though we are still young (March 15 - Ides of March) and only just now started to pick up some real steam.

Now if Slurp will just pick up the slack and get on with it -- I will be a happy camper all around.

sfowler

07-09-2004, 08:43 AM

Is it possible that the powers that be at Yahoo! have told slurp to concentrate solely on RSS feeds and the like for the moment? It would explain the total lack of Slurp activity anywhere else.

ronniethedodger

07-09-2004, 05:04 PM

Is it possible that the powers that be at Yahoo! have told slurp to concentrate solely on RSS feeds and the like for the moment? It would explain the total lack of Slurp activity anywhere else.

Slurp does not crawl those files, they just have a directory of them. The publisher of the RSS/Atom file will "ping" the directory to let it know that it is updated. It is up to the publishers to do this, and not Yahoo to run around and check.

greeneagle

07-09-2004, 06:14 PM

Inktomi is ahead of everyone else this month for me, slurping on it like an melting ice gream cone on a hot summer day.

This month so far:
Inktomi Slurp 40+37 561.56 KB 09 Jul 2004 - 18:58

Ken

ronniethedodger

07-10-2004, 08:46 AM

Inktomi is ahead of everyone else this month for me, slurping on it like an melting ice gream cone on a hot summer day.

This month so far:
Inktomi Slurp 40+37 561.56 KB 09 Jul 2004 - 18:58

Inktomi does not have a spider any more Ken. That is Slurp. Yahoo has not changed the designation for it on the nameservers. But if you look closely at email addresses in their record at WhoIs -- they will all say Slurp. Also, if you examine your raw log files, you will see that the User Agent field says Slurp as well.

Suggest you get a revision to your stats software to update the robots list.