Is Google Removing the Supplemental Index Tag?

[incrediblehelp]

A recent comment at SEOmoz.org from Matt Cutts says they might remove the SI tag sometime soon.

I guess Matt and Google were feeling that people are getting to "fixated" on their pages being in the SI. Well yes they are because these pages simply don't rank. They have absolutely no chance against pages outside of the SI. If they felt "fixation" was an issue why not take away PR number from the toolbar to that is so inaccurate. I know that will help me spend less time explaining what "truly" means.

Matt goes on to say:

"Over time, the supplemental results are less and less supplemental and more and more likely to show up for any given query. As I mentioned at SMX Seattle, my personal preference would be to drop the "Supplemental Result" tag altogether because those results are 1) getting fresher and fresher, and 2) starting to show up more and more often for regular web searches. Especially as the supplemental results get more fresh,"

More fresh? What the heck is that?

Micheal Martinez goes on to say this about "freshness":

Matt Cutts confirmed at SMX Advanced 2007 that Supplemental Results pages are not parsed the same way as pages in the Main Web Index. So if Google only "does away" with the "Supplemental Results" label, they will be playing a game of smoke and mirrors with both users and Webmasters.


Either Google should start fully parsing Supplemental Results pages so that they have a fair chance of ranking in search results, or they should keep the label so that Webmasters can see they still need to get more inbound links to move those pages into the Main Web Index.


Google should not under any circumstances pretend to be combining the two indexes. Either Supplemental Results go away completely or they stay. This is not a "freshness" issue. This is a "Google will not fully parse and index Supplemental Pages" issue.

So what does everyone think about this?

Here are some more source postings on the subject:

Update: Google is removing the supplemental query » Small Business SEM

Is Google Gearing Up To Drop The Supplemental Result Label?

[crankydave]

Well I can't say that I'm surprised but I don't like the idea of simply dropping the label.

Eary this year (i believe) Matt intimated that he thought the trend for 2007 would be for Google to start looking into the SI when they didn't find enough "good" matches in the RI for a query. Take a 3 word query for example. If Google was not able to find enough pages that contained all 3 words they'd then look into the SI for for pages that did, something that was not being done in the past. This is where the more and more SI pages showing up in regular search results comment is coming from IMO.

Additionally, as I have been watching, pages in the SI appear to be having less and less "value". I've noted that pulling a page out of the SI has had a dramatic effect on the page it is intended to target.

I said I'm not surprised because Google likely sees it at a means for webmasters to try and "manipulate" the pages that they want to have in the RI currently. This is adverse to what's been said about their thinking of what they want their "main index" to be. More comprehensive. More sites represented. Given that I'm of the firm belief that Google has decided to keep their forward index at a finite size, the supplemental label gives webmasters an opportunity to try and force more of "their" pages into the RI at the expense of other pages from other sites.

While I agree with MM that Google should fully parse and index pages in the SI so that they have a "better" chance of ranking, I don't see it as a "fair" chance since they look in the RI first and then the SI if they don't find enough "good" matches. A page in the SI may never even be considered for no other reason than it being in the SI to begin with. If Google were to take the best 20K matches from the RI and then the best 20K matches from the SI for their data set, then the pages in the SI would have a "fair" chance at ranking if they were fully parsed and indexed. To my knowledge, that's not being done.

Personally I see this move as a means for Google to take more control over the pages that are contained in the RI by not allowing webmasters and SEO's to see and target the ones that are not.

Dave

[Steven1976a]

What is the reason the SI appear anyway? Is it because of duplicate content? Or other factors involved?

Our new website is database driven and for a catagory like our necklaces we have 25 pages of necklaces which are exactly the same apart from the product pictures, stock number, prices.

If they all appeared in the serps then this would be bad for the serps right? They must filter out pages which are duplicate right?

[incrediblehelp]

Originally Posted by Steven1976a

If they all appeared in the serps then this would be bad for the serps right? They must filter out pages which are duplicate right?

true. I am not saying it is right or wrong to show people what pages are in the SI, but the issue is if you know your main pages are in the SI at least you know your doing something wrong.

Sure maybe me, you or Dave may not run into this, but for regular webmasters, knowing your pages are in the SI is a god reason to figure out where you went wrong.

[crankydave]

Originally Posted by incrediblehelp

true. I am not saying it is right or wrong to show people what pages are in the SI, but the issue is if you know your main pages are in the SI at least you know your doing something wrong.

Sure maybe me, you or Dave may not run into this, but for regular webmasters, knowing your pages are in the SI is a god reason to figure out where you went wrong.

Just to add to this, even if "duplicate" pages (folders) do make it into the RI index from the same site, take your product pages for example, they get filtered when the results sets are ordered. You might get an indented result for 2 pages but you won't get a whole list of them. They don't need the SI for that purpose.

You can do a search for something and use filter=0 and see what I mean.

Dave

[dvduval]

I think maybe they don't want to show webmasters which pages they don't like?

[tacimala]

From what I understood, they aren't removing the "tag" that says Supplemental Index next to the URL, they were simply removing the option to query just for supplement results using "site:domain.com **** -alskd". This command would show you all of your site's pages in the supplemental index and now it just gives you the same data as the normal site: command.

You can still run a site: command and go towards the end to find the pages tagged as a Supplement Result. The old command still works on some foreign Google TLD's though...

[laurynsmith]

When our rankings dropped dramatically, it was the supplemental index that lead us to find the answer why. We simply updated our (5 year old) code with do not follow and do not index tags to let Google know we were not spamming them or the user with useless information (it was photo galleries and former versions of updated pages that were supplemental). This was a valuable tool for us to make sure our code stays in compliance with Google. It will be a shame if they take it away from honest webmasters, just trying to keep content and code as up to date as possible.

[mpilatow]

Yeah Tacimala, they have already taken that functionality away. They are now talking about removing the Supplemental tag away completely. The reasoning according to Matt is that the supplemental index is being updated more often so it is more like the regular index. He also intimated that it would force SEO's to focus on traditional SEO techniques and not try to get their supplemental pages in the regular index since there is no supplemental index anymore.

I could see this going two ways. The first would be those supplemental pages get pulled into the regular index. The second would be those pages get dropped out of any index. In a way it would be preferable to have the pages be removed because at least you know there is a problem with them, which would help you figure out which pages need work. If they are in the index then you would need to do a bit more work to figure out what needs help. Either way still requires more investigation than the current method where you see exactly what pages are supplemental and need work.

[laurynsmith]

Sorry about that old signature tag everyone. Lauryn

[SemAdvance]

News Flash!


Google will only remove the label and the ability to find the pages in the supplemental index.

The supplemental index will not go away nor will your pages that reside within.

Just the tag leaves.....there always has been and will be two indexes

one for the few 1,000,000 pages that are somewhat worthwhile.....

and an index for the other 6,999,999,999 of worthless crap pages.

Nothing changes on the inside... just the fancy label on the packaging!!!

Edit:
Dave, for any page to be in either index, it would indeed had to have been indexed in the first place....could not get there any other way but through the indexing bot.

Peace

[webreporter]

Right? Wrong? What is Google now, God? Oh, wait a minute. I forgot where is was... The Church of the Google Faithful.

I have made dozens of pages according to Google standards, and have see countless other well made pages/sites that ended up in the supplemental index. Showing up in the main index too are pages chalk full of HTML mistakes and irrelevent content showing up high on Google searches. I have also seen them give high rank to pages that have "Ghost" HTML code being cached, not what you actually see show up in your browser.

Oops.

I have said this countless times before: Googles main and number one indexing criteria has to do with the popularity of the site and/or page. They have created what is tantamount to issues we faced in high school with the social scene. Just becuase they are popular doesn't make them right, or good.

Their system is flawed and will continue to be that way until the change the fundamentals of their criteria.

[benc007]

If the SI label is removed, there is no way of knowing which and how to get pages out of SI!

[SemAdvance]

Originally Posted by benc007

If the SI label is removed, there is no way of knowing which and how to get pages out of SI!

They have been telling webmasters for ages not to worry about the supplementals and yet most do...

So the way to get rid of the headache is to get rid of the label...

There has always been two indexes and will always be two indexes.


Reason

To return results in 100ths of a second they need to weed out the the good information from the useless.

No matter how much computing power they have they could never search through a billion pages fast enough to return the proper results in the time they do now.

[Fitz]

Originally Posted by benc007

If the SI label is removed, there is no way of knowing which and how to get pages out of SI!

Surely the SI label or index is not the real issue. Surely it's this simple: if your page doesn't rank, it needs further work (e.g. more IBLs). With the greatest respect, did we need a label to tell us that?

[crankydave]

Originally Posted by SemAdvance

Reason

To return results in 100ths of a second they need to weed out the the good information from the useless.

No matter how much computing power they have they could never search through a billion pages fast enough to return the proper results in the time they do now.

Perfectly useful and unique pages get placed into the SI all the time based upon nothing more than their, individual as a page and the site as a whole, link profile.

The supplemental index is not what it used to be... a repository for "unneeded pages". This changed more than a year ago with "Big Daddy".

Originally Posted by SemAdvance

Edit:
Dave, for any page to be in either index, it would indeed had to have been indexed in the first place....could not get there any other way but through the indexing bot.

I've not a clue what you're referring to. But if your're talking about pages in the SI not being completely parsed and indexed you might want to follow the links that Jaan posted and then you'll understand.

Dave

[Clarrie]

Originally Posted by Fitz

Surely the SI label or index is not the real issue. Surely it's this simple: if your page doesn't rank, it needs further work (e.g. more IBLs). With the greatest respect, did we need a label to tell us that?

And since when did IBLs ever reflect the qualititive value of a page? Never. I've always had an issue with the supplemental index as an entire concept, as I've seen too many quality pages (pages with useful, unique, worthwhile content) stuck in the SI because the site itself doesn't have enough "popularity". Typically these sites are serving a niche - either local services or a highly specialised market. Consequently they will NEVER get the load of IBLs needed to lift pages out of the SI, despite the quality of the pages themselves.

The SI is effectively a links based filter that favours the big, the branded, and the established regardless of content quality.

But if Google wish to persist with keeping the SI in existence, they should WITHOUT QUESTION continue to flag pages in the SI. To do anything else is underhanded and misleading.

One has to question why Matt / Google should be concerned about whether they are tagged or not - maybe its because they are increasingly concerned about the negative press (justified in my view) about the SI. If the SI is such a great thing, why worry about tagging pages. If they want to drop the tags, guess they must have something to hide.

[incrediblehelp]

And since when did IBLs ever reflect the qualititive value of a page? Never.

Pretty much always as Google is concerned. Sure I may not agree but what it is, is what it is.

I've always had an issue with the supplemental index as an entire concept, as I've seen too many quality pages (pages with useful, unique, worthwhile content) stuck in the SI because the site itself doesn't have enough "popularity".

Yup I agree, but most of that high quality content that is not ranking has not been marketed to it fullest extent,

Typically these sites are serving a niche - either local services or a highly specialized market. Consequently they will NEVER get the load of IBLs needed to lift pages out of the SI, despite the quality of the pages themselves.

Sure they will if they have the right marketer or if they are truly the serving to marketing the right niche.

[Clarrie]

Originally Posted by incrediblehelp

Pretty much always as Google is concerned

Quite so, and to a point it works well, but where it fails is when used for censoring, which is the effect of using popularity to determine which content is allowed in the main index, and which is in the SI. Popularity isn't always a good guide to quality - just take a look at at a few politicians!

Originally Posted by incrediblehelp

Yup I agree, but most of that high quality content that is not ranking has not been marketed to it fullest extent,

Sure they will if they have the right marketer or if they are truly the serving to marketing the right niche.

Time is Money, Money is heard. Quality + Money = Google Main Index? It's reality, but is it right?

Basically, helping these companies is what I do (and generally pretty well if given the resources) but some of these organisations don't have too many resources, and is that really a valid reason for being excluded from the main index of the leading search engine?

G should get rid of the SI and concentrate on improving relevance filters as an alternative. Maybe that's what Matt is trying to say, but they've a long way to go yet!