Supplementals: Not What They Used To Be & Tips for Getting Out

[mjtaylor]

I believe the hoo-ha over pages in the supplementals is based on an outdated concept. I have several sites which now have a larger proportion of pages in the SI than before; these are not duplicate content pages, but pages with similar listings, such as real estate listing detail pages and vacation rental listings. It makes sense to me that these pages are in the supplementals, as they generally have only one link to them from a menu page.

The greater proportion of pages in the SI has not affect my sites SERPs or traffic. I wouldn't expect those pages to come up for the more competitive terms, but some rank well for longer tailed terms, just as they did before.

I know this is 'old news', but since many webmasters still seem overly alarmed by pages in the SI, I think Matt Cutt's post last January bears excerpting (bold are my addition):

As a reminder, supplemental results aren’t something to be afraid of; I’ve got pages from my site in the supplemental results, for example. A complete software rewrite of the infrastructure for supplemental results launched in Summer o’ 2005, and the supplemental results continue to get fresher. Having urls in the supplemental results doesn’t mean that you have some sort of penalty at all; the main determinant of whether a url is in our main web index or in the supplemental index is PageRank. If you used to have pages in our main web index and now they’re in the supplemental results, a good hypothesis is that we might not be counting links to your pages with the same weight as we have in the past. The approach I’d recommend in that case is to use solid white-hat SEO to get high-quality links (e.g. editorially given by other sites on the basis of merit).

I think going forward, you’ll continue to see the supplemental results get even fresher, and website owners may see more traffic from their supplemental results pages. To check out the current freshness of the supplemental results, I grabbed 20 supplemental pages from my site and checked out their crawl date using the “cache:” command and looking in the cached page header. The oldest supplemental results page that I saw was from September 7th, 2006 (and I only saw 2-3 pages from September; most were from December or November). The most recent of the 20 pages was from January 7, 2007, which shows that supplemental results can be quite fresh at this point.

That said, I am not suggesting that having pages in the SI is a good thing, just that it is not the death knell that it was once believed to be. Here are some ways to get pages out of the SIs:

• ensure the content is unique
• add fresh content
• make sure all pages have unique title and meta description tags;
• obtain high quality, trusted links;
• restructure or create more internal links;
• link out to relevant authority sites;
• check for broken links;
• make sure you have no canonical issues.

Please feel free to add any tips for getting out of the supplementals ... or to disagree.

Cheers, MJ

[incrediblehelp]

My favorite reasons that pages go SI:

1. Exactly duplicated content
2. Over the top content/structure similarities
3. Pages with little or no content
4. Orphaned web pages. Pages that no one links to, including internal navigation
5. Poor website navigation
6. Pages due to canonicalization problems
7. Not enough quality back links
8. Long URLs, especially with long parameters, starting with a question mark (?) and being separated with an ampersand (&) and are not rewritten
9. Suspicious pages for spam-indexing, like non-unique and irrelevant to page content Meta tags, or linking to bad neighborhoods, etc.
10. Using duplicate description tags and title attribute across your website

So simply not doing these things will help keep you out.

[Palindrome]

Hi MJ

Thanks for the notification on this, good topic.

The Matt Cutts wording "I think going forward, you’ll continue to see the supplemental results get even fresher." was surely a Google hint. Unless many rational people are misreading their figures, a higher percentage of pages have recently hit the supplemental, regardless of other issues. I'm less sure about the "website owners may see more traffic from their supplemental results pages." I take your point about long tail searches but have been struggling lately to see many of these work, on quite a few sites.

In truth , I do not actually know whether calling up supplemental pages has become harder, just an impression. If anyone has recently collected specific data on this, that would be helpful. Particularly if that could be compared to historic information.

Must confess, I do prefer real data to impressions where possible and by coincidence, am currently running an experiment on getting pages out of the supplemental, with the kind help of a friend's site. We created a few pages at various levels of crudiness, knowing they'd go supplemental and have recently begun specific but different changes to each. Content, internal linking, IBLs etc. to see which is effective and with what speed.

However well you monitor site data, still hard to give exact opinion on which of many aspects were the telling factor but we'll hopefully get close. Should any clear information come out of this, I'll post an outline if that's of interest.

[Webnauts]

Originally Posted by Palindrome

In truth , I do not actually know whether calling up supplemental pages has become harder, just an impression. If anyone has recently collected specific data on this, that would be helpful. Particularly if that could be compared to historic information.

I think you kind find here what you are looking for: Big changes at Googles algorithms? Or did I miss something?

Read carefully the posts of the forum member BabyChicken4u2 there.

[Palindrome]

Thank you Webnauts, an interesting thread, with much to be gleaned.

Controlled IBLs, quality versus number etc. is part of what we are charting. Along with other options, both as additions to basic beliefs and in case they are wrong.

A little like the redundant tables, we'll never come up with an exact formula, in a field with so many variables. Even if we did, not sure about the lifetime of any benefits, can't escape a feeling that the rate of change within search engines is picking up.

[Seminole386]

We tend to think of one size fits all. I have real estate sites that generate 55 unique visitors per day. The supplimental pages generate 4-7 UV a week. But my average commission check is five figures and that is more than enough traffic for me. The main body pulls in buyers the supplimental pages pull in sellers. The main site is Florida Gulf Coast Condo Report- FL condos for sale. It would be of little value to think this way if I was selling $5 items but when your selling $500k and up its sufficient.

[espmartin]

Originally Posted by mjtaylor

...That said, I am not suggesting that having pages in the SI is a good thing, just that it is not the death knell that it was once believed to be. Here are some ways to get pages out of the SIs:

• ensure the content is unique
• add fresh content
• make sure all pages have unique title and meta description tags;
• obtain high quality, trusted links;
• restructure or create more internal links;
• link out to relevant authority sites;
• check for broken links;
• make sure you have no canonical issues.

Please feel free to add any tips for getting out of the supplementals ... or to disagree.

Cheers, MJ

MJ, your list can be titled, "What Every Website Author Should Be Doing" - or, "Web Designing 101"

I couldn't agree more...actually - how does one "agree more" anyway

[Webnauts]

Originally Posted by espmartin

MJ, your list can be titled, "What Every Website Author Should Be Doing" - or, "Web Designing 101"

I couldn't agree more...actually - how does one "agree more" anyway

Martin, IncrebibleHelp would confirm that I posted such a list about supplementals many times here the last two years, but never got credited that way. I am just wondering why?

Here is mine once again:

• Pages due to canonicalization problems, e.g duplicated content, too much content similarity;
• Pages with low or no content;
• Orphaned web pages. Pages that no one links to, including your own;
• Error pages, if a site does not use If-Modified-Since, Last Modified and/or Expires rules;
• Poor website navigation;
• Loading pages with irrelevant keywords (keyword stuffing);
• Too low PageRank;
• Long URLs, especially with long parameters, starting with a question mark (?) and being separated with an ampersand (&amp and are not rewriten;
• Suspicious pages for spamindexing, like non-unique and irrelevant to page content heading tags, meta tags, or linking to bad neighborhoods, etc;
• Too low PageRank.

Check how many of your pages are in the supplemental results with this tool: Supplemental Index Ratio Calculator

[incrediblehelp]

John is right, lots of my SI rules/items are from Johns list as well. He knows the SI to much, LOL

[Webnauts]

Originally Posted by mjtaylor

The greater proportion of pages in the SI has not affect my sites SERPs or traffic. I wouldn't expect those pages to come up for the more competitive terms, but some rank well for longer tailed terms, just as they did before.

It would be nice if you could back that up with evidence based facts (statistics), otherwise I consider what you are claiming not credible at all.

Originally Posted by mjtaylor

I know this is 'old news', but since many webmasters still seem overly alarmed by pages in the SI, I think Matt Cutt's post last January bears excerpting (bold are my addition)

This guy told us so many stories that after a while came out that were not true. And this story is one of them. If you look around at searchenginetable, digitalpoint, webmasterworld and other places, you will see that is not only my opinion. And thanks God for that.

Originally Posted by mjtaylor

Please feel free to add any tips for getting out of the supplementals ... or to disagree.

Cheers, MJ

I added my tips I am preaching here since I guess too years too. And I disagree that supplemental are not same or worse than before.
Google is more sensitive than ever before, so it is easier to go in the supplemental than ever before. And supplemental pages don't rank! A lot of evidence (statistics/numbers) are posted at CTabuk forums here: Big changes at Googles algorithms? Or did I miss something?

[Webnauts]

Originally Posted by incrediblehelp

John is right, lots of my SI rules/items are from Johns list as well. He knows the SI to much, LOL

Brother, I would not dare to claim that I am a guru or expert in SEO, but I would dare to call myself one for supplemental issues.

Supplemental results is also one of my hobbies.

[thindenim]

It's an interesting one this. As I've posted about elsewhere I've been having real problems with the SI recently, out of 1,300 indexed pages only 100 or so are currently in the main index. However: -

My traffic is actually up from what it was when I had 500 pages in the main index. I'm getting about 1,000 unique visitors a day now, and earlier in the year before the problems started I was getting around 700.

Part of this could be the Christmas factor, and the fact that there are just a greater number of people searching for my products. I gues I will have a clearer indication of this in January, when it may well be tumble weed central

The thing is though, it seems logical that if the pages were not supplimental then I would be getting more traffic as they would rank higher??? With this in mind I will continue in my efforts to get the pages back into the main index.

[Webnauts]

Originally Posted by thindenim

Part of this could be the Christmas factor, and the fact that there are just a greater number of people searching for my products. I gues I will have a clearer indication of this in January, when it may well be tumble weed central

I am sure someone will say that it is a Christmas factor. Wait and see.

[wige]

Originally Posted by Webnauts

It would be nice if you could back that up with evidence based facts (statistics), otherwise I consider what you are claiming not credible at all.

What sort of evidence would you like? My server log file is full of traffic from page 1 of Google's SERPS pointing to pages in the SI.

Originally Posted by Webnauts

Error pages, if a site does not use If-Modified-Since, Last Modified and/or Expires rules;

Are you saying that not using modification date information could cause SI issues?

Also, I have to wonder how accurate our calculations of Google's Supplemental Index size are today. Consider our method for determining what pages are in the main index, since SI pages are no longer marked - we pass a false query to Google that causes an internal error and causes some documents to not be included in the results. I have always suspected that the false query might only cause results from the current datacenter to be reported. This would block SI pages, since they are not in the same index, but if this is the case it would also block newly indexed, updated and modified pages that have not been pushed out to other data centers from showing up. This means that our calculations could, potentially, be inaccurate. Any ideas or suggestions how to test that the SI ratio calculation ONLY takes into account SI pages?

[Webnauts]

Originally Posted by wige

Any ideas or suggestions how to test that the SI ratio calculation ONLY takes into account SI pages?

1. Supplemental Index Ratio Calculator
2. How To Exploit The PageRankBot Tool » Half’s SEO Notebook
3. Manual checking

I was working on our site's supplementals today.

Here is what the one tool found: Supplemental Index Ratio Calculator

1 page in an https page on www.
8 pages are the categories of our forums which are duplicated due to redirect mistake:

To be specific, there two are indexed by Google:

http://forums.seoworkers.com/basic-search-engine-optimization/
http://forums.seoworkers.com/basic-search-engine-optimization-b10.0/

Any further questions?

We are now working on those and we will for sure be out of the SI sooner than you think.

We had more pages in the SI until 7-10 days ago and we were ranking steadily for the term "seo" between #112 and 114 and for "search engine optimization" # 193. Suddenly 2 days ago we jumped for the first #90 and the second one #156. And other less but high competitive terms improved dramatically. Is that because of Christmas time? And by the way we are talking about rankings and not traffic.

But I guess I do not need to show off or? If you think I should, I would glad to do so.

[wige]

Well, looking at my site in the tool, I have my lowest SI ratio ever, at a lowly 94.55%, with 1100 pages in the main index and a twenty thousand in the SI. A drastic improvement over last month, with no major changes to the site. So, yay.

Now, my question is, is there a way to ensure that the only difference between the results of the queries used to get these numbers is SI? For instance, do RSS feeds show up in one but not the other, and is there a way to determine if this is caused by an SI issue or the fact the page is not a normal web page? Is it possible pages that are detected as blogs or press releases don't show up in these queries? I know entire sections of my site that are not blogs or news stories show up as such in Google. They don't show up in the main index using the hack, but they do show up in blog search, and in the normal searches. Is this because they are in the supplemental index, or because they are in the blog index?

[wige]

I did some experimentations with everyone's favorite domain, mydomain.com. I was hoping that the site would have less than 1000 pages so I could test the accuracy of the numbers we all use to calculate the SI index, and it does. What I found was that the numbers are unreliable at best.

I ran the first query to get the total number of pages in google's index for the mydomain.com domain (site:mydomain.com) and got 894 results. I then did a query to get the number of pages in the main index (site:mydomain.com -fake:mydomain.com) and got 152 results. This, with some math, indicates that mydomain.com has an SI ratio of 83%.

Now, I checked those numbers for accuracy by seeing how many pages are actually indexed by Google. I ran my main query again, and this time used a hack to view the last "actual" indexed page (added &filter=0&start=893 to the URL) and found that Google only has 621 pages in the total index. This means that mydomain.com's actual SI ratio is actually 75%.

I did some additional testing, and have determined that the supplemental index not be the only index that is blocked by the -allinurl: parameter. I did a few searches that my site comes up for in Google's blog search, and checked the pages that were returned. I then entered these URLs into the filtered normal search and they were not returned as being in the main index. This could indicate that the blog index is seperate from the main index. I tested this further by searching for some of Matt Cutts' blog entries, and they showed up as not being in the main index either. Anyone interested in helping me test this further?

[espmartin]

Originally Posted by Webnauts

Martin, IncrebibleHelp would confirm that I posted such a list about supplementals many times here the last two years, but never got credited that way. I am just wondering why?

Here is mine once again:

• Pages due to canonicalization problems, e.g duplicated content, too much content similarity;
• Pages with low or no content;
• Orphaned web pages. Pages that no one links to, including your own;
• Error pages, if a site does not use If-Modified-Since, Last Modified and/or Expires rules;
• Poor website navigation;
• Loading pages with irrelevant keywords (keyword stuffing);
• Too low PageRank;
• Long URLs, especially with long parameters, starting with a question mark (?) and being separated with an ampersand (&amp and are not rewriten;
• Suspicious pages for spamindexing, like non-unique and irrelevant to page content heading tags, meta tags, or linking to bad neighborhoods, etc;
• Too low PageRank.

Check how many of your pages are in the supplemental results with this tool: Supplemental Index Ratio Calculator

Here here Webnauts, I do recognize that your list is very good, and can be considered a "Web Design 101" list
as well.

Hey - you gotta share the love!!!

[Webnauts]

Originally Posted by espmartin

Hey - you gotta share the love!!!

I would have done that, if people would just trust me after all, and would not try to turn down or doubt things I am saying, since I only tell things when I am damn sure about. And have no time for debating as I had before.

I just want one member in this forum to tell when have I gave him/her a tip and it did not work or have been damaged.

Anyway... forget it man.

[espmartin]

Originally Posted by Webnauts

...if people would just trust me after all...

Anyway... forget it man.

Well, I have read your comments on this forum, as well as your own site/forum/blog - and I have to say that I have
agreed with everything you have said (from what I have read so far )

[Webnauts]

Originally Posted by espmartin

Well, I have read your comments on this forum, as well as your own site/forum/blog - and I have to say that I have
agreed with everything you have said (from what I have read so far )

Martin, could it be that you are the only one?

[Halfdeck]

A couple other factors Googlers mentioned:

- URL complexity
- Page freshness

[Webnauts]

Originally Posted by Halfdeck

A couple other factors Googlers mentioned:

- URL complexity
- Page freshness

Page freshness? Can you tell us where you read that?

[wige]

Originally Posted by Webnauts

I would have done that, if people would just trust me after all, and would not try to turn down or doubt things I am saying, since I only tell things when I am damn sure about. And have no time for debating as I had before.

Honestly, it has nothing to do with trust, or doubting the accuracy of your statements, at least for me personally. Its just that I am part hacker, and there is a bit of a philosophy to that side of me: The only way to learn anything, is to question everything.

[Webnauts]

Originally Posted by wige

Honestly, it has nothing to do with trust, or doubting the accuracy of your statements, at least for me personally. Its just that I am part hacker, and there is a bit of a philosophy to that side of me: The only way to learn anything, is to question everything.

I only confirm what is already known Wige.

[Webnauts]

I felt like posting this information, which clarifies pretty well the affects of the supplemental pages:
Got Supplementals? Accepting PageRank is Only The Beginning » Half’s SEO Notebook
and
Google Dumps Supplemental Name Not Index

[ldylion214]

Originally Posted by incrediblehelp

My favorite reasons that pages go SI:

1. Exactly duplicated content
2. Over the top content/structure similarities
3. Pages with little or no content
4. Orphaned web pages. Pages that no one links to, including internal navigation
5. Poor website navigation
6. Pages due to canonicalization problems
7. Not enough quality back links
8. Long URLs, especially with long parameters, starting with a question mark (?) and being separated with an ampersand (&) and are not rewritten
9. Suspicious pages for spam-indexing, like non-unique and irrelevant to page content Meta tags, or linking to bad neighborhoods, etc.
10. Using duplicate description tags and title attribute across your website

So simply not doing these things will help keep you out.

I have a lot of these problems but I'm working to correct them. I notice everyday I have more in SI and less in main. Yesterday I made a robot.txt file including all my showitem.asp. Each one of those will be duplicates in the eyes of the SE. My question is how long will it take Google to no longer list those pages as SI so that my ratio isn't 95%. I also cleared alot of useless pages off my website. These were pages no one linked to anymore. I didn't realize those would cause issues if there was no link from my site to them.
I'm taking baby steps but I am trying. Seems the more I do the worse it gets but I think that is because I was clueless before I started picking the gurus brains here.

Best, Nicci

[Webnauts]

Originally Posted by ldylion214

I have a lot of these problems but I'm working to correct them. I notice everyday I have more in SI and less in main. Yesterday I made a robot.txt file including all my showitem.asp. Each one of those will be duplicates in the eyes of the SE. My question is how long will it take Google to no longer list those pages as SI so that my ratio isn't 95%. I also cleared alot of useless pages off my website. These were pages no one linked to anymore. I didn't realize those would cause issues if there was no link from my site to them.
I'm taking baby steps but I am trying. Seems the more I do the worse it gets but I think that is because I was clueless before I started picking the gurus brains here.

Best, Nicci

I am dealing already with some sites and I am facing the same issue. No need to worry. Pages that had to be in the main index have been in the SI and vice versa.

The issues you see will be solved when Google will update the SI. And I noticed so far that takes place once a month so far.

So stay cool.