Supplemental club

[Ellio]

No sign of recovery yet.

Not sure what the rest of Kens post was on about.

[SPC2]

Originally Posted by crankydave

If what you are saying were correct, then they would be storing the information they fetch from the fully indexed page and the partially indexed page in the same index. In another words, two sets of information, two exactly identical URL's, for the same page stored in the same barrel. Makes no sense.

It makes sense to me, Dave. The short barrels contain words that are found in such things as link text and page Titles. They also contain data as to which URLs those words apply to. So Google finds a link with link text and a URL, and it stores them in one of two places:- (1) in the short barrels, or (2) somewhere else. Later, they fetch the URL's page, parse it, and store its Title words in the short barrels, along with data about its URL.

Then they do one of two things:- (1) they go and get the previous link text words and pointers, and store them in the short barrels, because they now have the page itself, or (2) they do nothing, because that data is already in the short barrels.

I see no reason NOT to put the link data in the short barrels straight away, and I see good reason to put it there straight away.

Some circumstantial evidence is that we sometimes see a URL-only listing in the normal serps, when Supplementals have not been included. That could happen because the link text data alone (in the short barrels) is sufficient to get the URL ranked.

[SPC2]

Originally Posted by Pascal77

Some people say they are out of supplemental. Then my question is : and how about rankings and visitors?

My rankings are back to normal. They disappeared while the site's pages were supplemental, but all is well now.

[Pascal77]

Originally Posted by SPC2

My rankings are back to normal. They disappeared while the site's pages were supplemental, but all is well now.

Maybe it will happen here a bit later but i don't see any recover.

Are there a lot of sites which have their rankings back?

[SPC2]

Addition to my reply to Dave a few posts back...

Remember that the barrels contain individual words, together with some data about each word - they don't contain bunches of text from pages or from links. Also remember that the words in links texts and the words in a page are different things - they are usually on different pages, even though the URL (page) that they are associated with is the same. So no duplication occurs in the short barrels if they store the link text words in them, and then later store the words from the page in them.

The only difference between a URL-only listing, and a listing that is based solely on link text (e.g. miserable failure) is that Google has the page that is associated with the miserable failure listing, but they don't have it for the URL-only listing. I.e. the queries are handled identically, and then the programme checks to see if they have a page or not for the URL. If yes, then get a snippet; if no, then only show the URL. That's the way I see it, anyway.

[SPC2]

Pascal77. Have you done what GG suggested, and contacted them about it?

[Ellio]

Posts in this thread keep going missing.

Could the administrator doing this please explain.

[Pascal77]

Originally Posted by SPC2

Pascal77. Have you done what GG suggested, and contacted them about it?

Yes, don't count on that really

Dutch site, young site, ...
I don't think they are going to spend much time about that. No problem, this has brought to my attention that i should optimise for other SE's also(Google has 85% here) and hope that they win popularity :-) .
I learned a lot with this Gproblem, started experienting with adwords, ... So it's not al bad but i prefered to see my site grew. I was planning to go for only organic results and see how long it takes to get results and at whiwh time you get which results. So it was something to learn but i think the learning stage is over. This has a to big effect to take as example of how a site grows natural. But Google has lost my faith here and that never happened before. That is something that is going to play when i make future decissions about advertising and all. I wil also look at other players now.

[greeneagle]

Dave,

I reallly believe that the associated issues here surrounding on-line stores right now are an important and timely issue, so I broke it off over here:

http://www.webproworld.com/viewtopic.php?t=61723

Ken

[crankydave]

Originally Posted by SPC2

It makes sense to me, Dave. The short barrels contain words that are found in such things as link text and page Titles. They also contain data as to which URLs those words apply to. So Google finds a link with link text and a URL, and it stores them in one of two places:- (1) in the short barrels, or (2) somewhere else. Later, they fetch the URL's page, parse it, and store its Title words in the short barrels, along with data about its URL.

Then they do one of two things:- (1) they go and get the previous link text words and pointers, and store them in the short barrels, because they now have the page itself, or (2) they do nothing, because that data is already in the short barrels.

I see no reason NOT to put the link data in the short barrels straight away, and I see good reason to put it there straight away.

Some circumstantial evidence is that we sometimes see a URL-only listing in the normal serps, when Supplementals have not been included. That could happen because the link text data alone (in the short barrels) is sufficient to get the URL ranked.

Phil,

Take a look at my URL only result on page 18 of one of the affected DC's...

http://216.239.59.99/search?q=%22St....&sa=N&filter=0

Every single result for this search, all 18 pages, is marked supplemental except my URL only partially indexed result.

Now what do you think?

Dave

[SPC2]

I can see what you are getting at, Dave. But it's not the last result, and non-supplementals don't always outrank supplementals.

I've explained why I see no reason to store a link's data in a different index to where it will be when the target page is fetched and parsed (it will end up there anyway, and the data is useable for rankings without the associated page's data), and your example doesn't change my thinking. I can suggest a reason why the results are as they are ...

Since a query in the short and long barrels only returned one result (yours), they went to the supplemental index to try and find more. That's the normal way that they do things. Having found more, they then ranked the results set, and returned them in the order that we see.

[Pascal77]

I find it very strange that Google keeps silence about this.

There's something very weird with this case.
I also don't find any info in any SEOnewsletter. And that after a week of heavy problems.(Why? Isn't it important enough?)

Is there any clue on how much sites are affected and how many recovered? Are it just a few sites and why just those sites?
Answering to these questions is what Google must do, correction, what they had to.

[crankydave]

Originally Posted by SPC2

I can see what you are getting at, Dave. But it's not the last result, and non-supplementals don't always outrank supplementals.

I've explained why I see no reason to store a link's data in a different index to where it will be when the target page is fetched and parsed (it will end up there anyway), and your example doesn't change my thinking. I can suggest a reason why the results are as they are ...

Since a query in the short and long barrels only returned one result (yours), they went to the supplemental index to try and find more. That's the normal way that they do things. Having found more, they then ranked the results set, and returned them in the order that we see.

There are 2 non supplemental results on page 1 that I see now. 1 for my site, and one from a competitor. There are 169 returns marked supplemental, and one URL partially indexed page.

Guess we can agree to disagree if you're insisting that my URL only result is in the regular index.

Dave

[SPC2]

I'm not insisting on it. I'm only saying that it makes good sense to me to put the data in the short barrels, and poor sense to me to put it elsewhere. I can't say that I'm right about it, because there may be plenty of things that I haven't considered. I'm only seeing it as an overview.

I can't see the data being in the supplemental index though - else why isn't the listing marked as supplemental.

[crankydave]

Originally Posted by SPC2

I'm not insisting on it. I'm only saying that it makes good sense to me to put the data in the short barrels, and poor sense to me to put it elsewhere. I can't say that I'm right about it, because there may be plenty of things that I haven't considered. I'm only seeing it as an overview.

I can't see the data being in the supplemental index though - else why isn't the listing marked as supplemental.

Same reason it doesn't show a cache. Nothing to mark no data. Only a URL they found but had yet to visit, so they don't even know if the page exists yet. I do see your point on initially, where's the best place to put it. But, once they go back to that URL and index and cache the page, they're not going to leave the the URL only page in the same place as the newly indexed page.

Nearly 2 years ago they found the URL. Shortly after finding the URL they went and fully indexed that page. No reason for them to keep the URL only partially indexed page anymore now that they have fresh data. They're not going to throw it away, and keeping that page in the same index as the newly crawled page makes no sense. If the URL only was in the regular index, then doing a search for it would pull it up on one of the unaffected DC's. Guess what... it does not. Why? Because they have the fully indexed page showing as a regular result.

Dave

[SPC2]

Just because the page hasn't been crawled, parsed and stored, doesn't mean that the listing can't be marked as supplemental in the serps, when it's been acquired from the supplemental index. There's nothing preventing them from including the words "Supplemental result" with a URL-only listing.

You seem to have overlooked something, Dave. You are still talking as though the partially indexed data and the crawled page's data are the same things - at least in part. But they aren't. They are completely different things, and completely different data. In other words, the data that they have for the URL-only listing is not only different from the data that they have when the page has been crawled and parsed, but it's also needed in addition to the page's data, AND it is stored in the short barrels along with any data from the page that they also store in the short barrels (e.g. the words in the Title).

The data for the URL-only listing consists of the words that were found in the link text on a different page, and the target URL. They don't come from the URL's page at all. So when the target URL's page is fetched and parsed, there is no duplication of data whatsoever. Both lots of data are needed, and both the link text (URL-only) data and some of the page's data, are stored in the short barrels.

That's why I say that it makes very good sense to store the URL-only data in the short barrels as soon as they get it. It's going to finish up there anyway (it's never going to be thrown away or moved away - it's the very backbone of their ranking algo), and in the meantime it can be used for rankings without the page itself needing to be fetched and stored.

The URL-only data is the link text data that Google bases their rankings on to such a great extent. The data from the page is the content data, and it's completely different from the link text data. They acquire both lots of data at different times, and there's no duplicated data to move to someplace else. You are assuming that there's an overlap of data, but there isn't an overlap. One is link text for the URL, and the other is content for the URL.

[Ellio]

Anybody seen any further changes:

Sites back from hell?

Sites getting worse?

Other changes?

[Pascal77]

Originally Posted by Ellio

Anybody seen any further changes:

Sites back from hell?

Sites getting worse?

Other changes?

surely it's not getting better overhere

[janeth]

Has anyone herd anything else from Google.

Maybe there happy with the results?

[crankydave]

Phil,

No, there's nothing preventing them from marking the listing as "supplemental", I'm suggesting that they are not doing it.

Prior to this mess, the only page that could be found was the fully indexed page. You could not find the URL only page. This tells me that the URL page was not in either the short or the long barrel as a separate page otherwise it would have showed up in a search.

Currently, in the affected data centers, you can only find the URL only listing. You cannot find the fully indexed page. This tells me that the fully indexed page was not dumped into the supplemental index but rather had been lost or the data could not be retrieved.

Currently, in the repaired DC's, the only page that shows up is the fully indexed page. You cannot find the URL only page.

Finally, here's a quote from GoogleGuy...

Hey, pages get added to the supplemental index using automatic algorithms. You can imagine a lot of useful criteria, including that we saw a url during the main crawl but didn't have a have a chance to crawl it when we first saw it.
Think of this as icing on the cake. If there's an obscure search, we're willing to do extra work with this new experimental feature to turn up more results. the net outcome is more search results for people doing power searches.

Dave

[Pascal77]

Originally Posted by janeth

Has anyone herd anything else from Google.

Maybe there happy with the results?

But why don't they say this if this is the case?

If they are happy with this then i think that they are going to loose marketshare. SERP's are less relevant then before and this started in februari.

They have blown it in my eyes, the longer they wait to make it clear, the longer it's going to take for recovery.
Perhaps they lost the battle and have to start all over

[SPC2]

I could discuss alternative interpretations about the rest of your post with you, Dave, but the most interesting part is the GG quote. How old is it?

The quote makes it sound like the data (link text) from uncrawled URLs really is stored in supplemental, and I can't argue with that. The only thing I can do now is notice when a URL-only listing is shown in the serps to see if other supplementals are also included, or not. If no others are included, I'd still go with my theory, in spite of GG's comment, but if they are, I'd have to abandon my theory.

Either way, it still makes much more sense to me to store the URL-only data in the short index from the start (it's never stored in the long index), because it has to be copied to there at some stage, or the links-based algo won't work.

[SPC2]

I just had a thought.

Google has talked about the Supplemental index as being part of their Auxiliary index. I wonder if another part of the Auxiliary index is an index where URL-only data is stored until the URL's content is fetched and indexed. How's that for a spanner in the works? :-)

[crankydave]

Originally Posted by SPC2

I could discuss alternative conclusions about the rest of your post with you, Dave, but the most interesting part is the GG quote. How old is it?

The quote makes it sound like the data (link text) from uncrawled URLs really is stored in supplemental, and I can't argue with that. The only thing I can do now is notice when a URL-only listing is shown in the serps to see if other supplementals are also included, or not. If no others are included, I'd still go with my theory, in spite of GG's comment, but if they are, I'd have to abandon my theory.

Either way, it still makes much more sense to me to store the URL-only data in the short index from the start (it's never stored in the long index), because it has to be copied to there at some stage, or the links-based algo won't work.

August of 2003 when the supplemental index was introduced. The URL we've been talking about would have been initially crawled within 6-7 months of that quote.

Now, it is possible for a URL only listing to be in the regular index. The thing to note is when you see a URL only result in the SERP's if there is a cache. This is an indication that the URL has been previously crawled but on a subsequent crawl there were unable to retrieve any data. In this is the case, then it would be in the regular index.

In the case that we have been discussing regarding my site, it's still my belief that every one of those URL only results are in the supplemental index and directly related to the problem Google is having right now. Too many things point to this being true to think otherwise.

If I am indeed correct, I believe I am, then none of the pages have been "moved" to the supplemental index. They are pages that have been there all along and the pages in the regular index have either been lost or they are unable to be retrieved because of this problem.

Since when does Google makes sense? :)

Dave

[crankydave]

Originally Posted by SPC2

I just had a thought.

Google has talked about the Supplemental index as being part of their Auxiliary index. I wonder if another part of the Auxiliary index is an index where URL-only data is stored until the URL's content is fetched and indexed. How's that for a spanner in the works? :-)

A very interesting thought. It would make more sense to place the URL only data together instead of mixing it in with the rest. Kinda like a "to-do-list".

Dave

[crankydave]

To follow up, how likely do you think it is that a site has zero pages in the supplemental index?

Dave

[janeth]

Originally Posted by Pascal77

But why don't they say this if this is the case?

If they are happy with this then i think that they are going to loose marketshare. SERP's are less relevant then before and this started in februari.

They have blown it in my eyes, the longer they wait to make it clear, the longer it's going to take for recovery.
Perhaps they lost the battle and have to start all over

I'm not sure that they have anything to gain by telling everyone there happy and leaving things as they are.

But they have stopped talking about it, sounds like to me there leaving it as is.

Has anyone seen anything change since they said they had fixed the problem?

[SPC2]

Originally Posted by crankydave

Since when does Google makes sense? :)

To us or to them??? :)

[Pascal77]

Originally Posted by janeth

I'm not sure that they have anything to gain by telling everyone there happy and leaving things as they are.

But they have stopped talking about it, sounds like to me there leaving it as is.

Has anyone seen anything change since they said they had fixed the problem?

Maybe they were aware that this could happen. Maybe they knew it would happen.

Who wil tell?

[Ellio]

Why ask for feedback from those still effected if they know what the problem is.

They are obviously as confused as we are.

However they know their algos better than we do so lets give them just a little more time to fix it all before storming the castle....

[crankydave]

Originally Posted by Ellio

Why ask for feedback from those still effected if they know what the problem is.

They are obviously as confused as we are.

However they know their algos better than we do so lets give them just a little more time to fix it all before storming the castle....

Storming the castle sounds like more fun! (insert evil smiley here)

Dave

[Ellio]

Well Dave,

You do have a point!

[Ellio]

I have received a reply from Matt Cutts via his blog:

Gary and Rahul #26: I’m looking into it. Lots of people have come back, and I’m asking someone to read the stillsupplemental emails from WMW.

Hopefully a fix may be on its way.

[janeth]

Originally Posted by Ellio

I have received a reply from Matt Cutts via his blog:

Hopefully a fix may be on its way.

It does sound like enough people are asking that it should make them look into it.

But like you said before it also sounds like they do not have a clue what is causing the problem.

[crankydave]

Originally Posted by Ellio

I have received a reply from Matt Cutts via his blog:

Hopefully a fix may be on its way.

Good! (insert optimistic clapping smiley here)

Dave

[janeth]

We are looking into it can also mean, just go on and leave me alone.

[SPC2]

My strong impression of Matt is that he is good to his word.

[Ellio]

I tend to agree I am confident that he will at least try to get this mess sorted.

Whether he is sucessful or not is another matter.

[AndrewX]

Reminds me of the good ole' "Google is Broken" days.

I will be excited to see this "fixed". My experience historically is that Sups are pages on the way out of the G Index, not nice to see your nice DMOZ listed site sitting in mostly in Supplemental Land. :rolleyes:

[williamc]

Andrew, i would agree 100% on not wishing to see my sites there.

[greeneagle]

williamc,

Seems like "gray" usually turns "black"...

Just another greeneagle philosophical "rant"!

There's good "medicine" here:

http://www.seo4fun.com/notes/supplementals.html

IMO - I believe they have an antedote, for some.

Ken

[williamc]

Once again Ken, you pasted that before, that has nothing to do with the current google screw-up. Those are the old regular things about the supplemental index. I have actually gone thru it and compared it to some sites still messed up and see no corelation.

[janeth]

Has anyone found any corelation with the sites that are in the supplemental index?

[crankydave]

Originally Posted by janeth

Has anyone found any corelation with the sites that are in the supplemental index?

Janeth, with the few Ive looked at, I've not noticed or found a commonality between the sites themselves.

What I suspect the common thread may be is a particular time period when sites were being crawled, reindexed, etc., or perhaps a particular bot, and the data that was being retrieved was unusable. The data in the main index was replaced with the unusable data. This lead to them having to use the supplemental results.

Dave

[Pascal77]

Originally Posted by crankydave

Janeth, with the few Ive looked at, I've not noticed or found a commonality between the sites themselves.

What I suspect the common thread may be is a particular time period when sites were being crawled, reindexed, etc., or perhaps a particular bot, and the data that was being retrieved was unusable. The data in the main index was replaced with the unusable data. This lead to them having to use the supplemental results.

Dave

Could it be a solution to forbid Googlebot for a few days and then reallow it?

I have no idea what effect it has on the bot when i deny it to crawl my site otherwise i tried it.
If you deny acces for a few houres and then reallow it, will the bot then return immediately?

[janeth]

But would'nt it be fixed on the next crawl?

[Pascal77]

Originally Posted by janeth

But would'nt it be fixed on the next crawl?

Perhaps i get a better bot then because now he is only swinging in my forum (Einstein face, :-) )

[Ellio]

Can I get back to the thought that this issue is really about the loss of "good" pages from the index leaving only the "Good" homepage.

I think the supplemental pages are a bit of a red herring as in our cases most of them should be there as they are "bad" (old, orphaned, deleted etc) and were probably there all along.

They have been simply hidden from view as there were previously more important pages to list before they dissapeared.

The issue is where have the "good" pages gone and when are they coming home!

PS: Love the "greeneagle effect" gauge.

[SPC2]

Assuming that the problems we see/saw are the same, then the pages *were* being put in the supplemental index - good pages, et al.

I can't see it being a crawling or bot problem, Dave, because the homepages weren't supplementaled along with the rest.

[Pascal77]

Originally Posted by SPC2

Assuming that the problems we see/saw are the same, then the pages *were* being put in the supplemental index - good pages, et al.

I can't see it being a crawling or bot problem, Dave, because the homepages weren't supplementaled along with the rest.

I was just checking sitemaps.
Since a week it is not visited. Last time i checked it was at least 2 times a day.

Somebody else who have this?