Any ideas why this page is supplemental?

[Webnauts]

Originally Posted by thindenim

Thanks for your help, I'll let you know if there is any change

Great! Be sure you took a wise decision. Keep me up-to-date, so I can see what else can be done next if necessary.

[thindenim]

Thanks again for your help.

I've actually taken this a step further and added

User-agent: *
Disallow: /shop/*/*/*/

which will stop my sub category pages being indexed. As you can browse to products from the category pages above these, they are really just creating duplicate content and I guess diluting pagerank unncecessarily.

I'm also going through seeing which other pages are not really required to be indexed. Hopefully this will start to make a difference.

[wige]

Originally Posted by Webnauts

User-agent: *
Allow: /*?$
Disallow: /*?

That will eliminate the cart pages. You don't need them. And you are waisting valuable PR and Rankings. You can take my word for that!!!

Would a robots.txt entry such as above have an effect on the distribution of internal pagerank? My understanding was that the weight of links was calculated when the spider arrived on the linking page, rather than when the linked-to page is crawled, and the robots.txt file is processed when the linked-to page is about to be requested.

In other words, if you have two links, each receives 1/2 of the available wieght. If one link is blocked by robots.txt, that link does not receive pagerank because it can't be crawled, but Googlebot doesn't know this when it calculates the rank, so the other link still only receives 1/2 the benefit. If you use nofollow, however, Googlebot knows right away that it can't go to the second link, so the first link gets only full value.

[Webnauts]

Originally Posted by wige

Would a robots.txt entry such as above have an effect on the distribution of internal pagerank? My understanding was that the weight of links was calculated when the spider arrived on the linking page, rather than when the linked-to page is crawled, and the robots.txt file is processed when the linked-to page is about to be requested.

In other words, if you have two links, each receives 1/2 of the available wieght. If one link is blocked by robots.txt, that link does not receive pagerank because it can't be crawled, but Googlebot doesn't know this when it calculates the rank, so the other link still only receives 1/2 the benefit. If you use nofollow, however, Googlebot knows right away that it can't go to the second link, so the first link gets only full value.

At some point I do agree with you. But not all the way.
I am aware of the option of using the nofollow attribute for shopping carts, but I do not see that as a best alternative in long terms. The nofollow tag does not prevent Google of indexing pages. So I prefer to have control over Google and not Google over me. What am I telling Google using the nofollow tag? That they should not trust or give value to all those pages of my site?

Since we are talking about duplicated pages, is the PR shared within those documents? No.

And if the pages being deleted which they only had PR from other internal pages, why should the PR get lost? The internal pages linking to those deleted pages will get the shared PR back. Or did I miss something?

[activeco]

Originally Posted by Webnauts

The nofollow tag does not prevent Google of indexing pages.

Yes, it does.
Let me repeat my post from another forum:
================================================== ====
Rel="nofollow" behaves in the exactly the same way as "nofollow" meta tag, which means the linked pages will not be crawled at all.

So, if that is the only link on the web, the linked page will never be crawled.
The problem arises when the page is normaly linked from some other place, where the bot gets the green light for crawling, unless the indexing of the linked page is forbidden in robots.txt or in meta's using "noindex" attribute.
================================================== ====

I have thoroughly tested this since March when I put a few unique pages linked with 'rel=nofollow' (and linked from nowhere else).
Results:
- Anchor name used for linking is shown as the property of the original linking page.
- The name of the linked file doesn't exist in Google index.
- No unique text in linked file can be found in Google.

[Webnauts]

Acticeco just another question before we go on. Will the pages which are already indexed be de-indexed after you add the no-follow attribute in the links pointing to those pages?

[activeco]

Originally Posted by Webnauts

Acticeco just another question before we go on. Will the pages which are already indexed be de-indexed after you add the no-follow attribute in the links pointing to those pages?

An interesting question.
I don't think so, but the only way to find it is to try it out. But remember that we are talking about documents linked ONLY and exclusively with 'rel="nofollow"'. If the document is regulary linked from somewhere else then the test is broken, so even if the outcome of the test is positive it will be almost of no use in the real world as you can't control natural linking to your pages.

[incrediblehelp]

Originally Posted by activeco

Rel="nofollow" behaves in the exactly the same way as "nofollow" meta tag, which means the linked pages will not be crawled at all.

What do you think about this thread?

MasterLink Search Marketing: "nofollow" - Does it Really Work Like Google Claims?

[crankydave]

Originally Posted by Webnauts

Acticeco just another question before we go on. Will the pages which are already indexed be de-indexed after you add the no-follow attribute in the links pointing to those pages?

Provided the only links to the page have had the nofollow attribute added to them, yes. The page becomes orphaned. Orphaning was also brought up as something that can be caused by using nofollow by MC.

Stands to reason that if a page can be orphaned by the nofollow, then the links are not followed.

Dave

[incrediblehelp]

I don’t feel the nofollow is not a do not crawl directive. It supposedly a do not pass juice directive. Googlebot doesn’t go through a website and look at attributes when crawling. They bring all the data back to the googlebot servers for analysis. Analysis is not done "on the fly". Now once Google crawls your website and sees the nofollow on a link, eventually they may take it off their crawler schedule.

[activeco]

Originally Posted by incrediblehelp

I don’t feel the nofollow is not a do not crawl directive.

No need for feelings, it's very easy to test.
'Nofollow' historically means 'no crawl' and it remained the same.

[crankydave]

Originally Posted by incrediblehelp

I don’t feel the nofollow is not a do not crawl directive.

If this were true, why would MC suggest a page could be orphaned by using it?

Anybody who uses nofollow "to modify PageRank flow at link-level granularity" (luv that engineer talk), must be absolutely sure it doesn't inadvertently orphan any important pages.

And...

There's no stigma to using nofollow, even on your own internal links; for Google, nofollow'ed links are dropped out of our link graph; we don't even use such links for discovery

Matt Cutts Provides Insight into the "nofollow" Attribute

Here's what MM states...

You could drop all your Supplemental Pages from the index by Nofollowing them (assuming no one else is linking to them).

Why Rand Fishkin’s nofollow post was wrong - SEO Theory - SEO Theory and Analysis Blog

Dave

[incrediblehelp]

I guess you both are missing the point. If a page has one back link to it with nofollow (internal or not) and 10 without it, the page is GOING to get crawled.

[crankydave]

Originally Posted by incrediblehelp

I guess you both are missing the point. If a page has one back link to it with nofollow (internal or not) and 10 without it, the page is GOING to get crawled.

I understand this Jaan.

The nofollow attribute is attached to a link, not a page. It tells a bot do not crawl the target page from this link. It is a link directive not a page directive.

Dave

[onlydo]

this can occured acording to the search term.Because Google will rank the websites whose content is most closely on topic for the search term you are looking for. Such website ranking is not entirely based on the content being on topic – other factors such as back-linking will also determine the ranking – but the content makes up a large part of the ranking algorithm.