The Power of the WebProWorld SEO Community

[kgun]

Originally Posted by kgun

If a person on the white list links to a page in the protected area (why should (s)he want to do that? - On an assumption that the protected area will be allowed later?) from hers / his site with a semantic anchor text, the link may be indexed, but it is still broken (dead). How probable is such a broken link that degrades that external page?

A follow up on that idea. Assume that,

1. The webmaster's ISP IP is on the white list and
2. his site is hosted by a hoster on the white list

then (s)he will not see the link as broken. The not white listed SE will see the anchor text but are unable to crawl and index / archive the page(s in the protected area).

I thnk that is not problematic for Google as reported in Matt Cutt's video or am I wrong?

[Terry Van Horne]

Originally Posted by Webnauts

Thank you for the link. I already know this story, but I am not wondering who is right: Google passes second link’s anchor text | SEO Theory - SEO Theory and Analysis Blog

I am still wondering if I have 3 links to the same destination, won't their power be weaken??

John, not to be facetious but why do you care? If there is a good reason to link that many times... then... I assume it's good for the user and that is what really counts.

[kevsta]

Originally Posted by Webnauts

So I am wondering if the second and third link is ignored by Google, then they do not need to be nofollowed, or?

IMO, to be sure of the best ranking available they don't need to be nofollowed, they probably need to not be there at all.

nofollowing them may or may not help things depending on the individual site's layout and format but in reality that's always going to be "suck it and see" guesswork.

[crankydave]

Just briefly... no pages are unimportant IMO. Some are more important than others. Rather than try and granularly funnel bots, why not use the pages you consider less important to your advantage?

Slight alterations to navigation, anchors, etc. comes to mind.

Dave

[kgun]

Originally Posted by kgun

A follow up on that idea. Assume that,

1. The webmaster's ISP IP is on the white list and
2. his site is hosted by a hoster on the white list

then (s)he will not see the link as broken. The not white listed SE will see the anchor text but are unable to crawl and index / archive the page(s in the protected area).

I thnk that is not problematic for Google as reported in Matt Cutt's video or am I wrong?

I think I must have been tired.

It is of course the IP of the webmaster's ISP that is relevant. The link to the page in the protected region can be on a server outside the white listed region. (S)he will still see the page if (s)he clicks the link from a computer with Internet access in the white listed region.

But there may be bots in the white listed Ip region and then, by replication, the content is potentially open to the whole world as far as I can see.

Originally Posted by crankydave

Just briefly... no pages are unimportant IMO. Some are more important than others. Rather than try and granularly funnel bots, why not use the pages you consider less important to your advantage?

There may be (educational or other) content that you don't want to be publicly available.

Conclusion:

If you white list Ip's (Ip regions), you must be careful that there is no bot on that list (in the Ip region). As long as we are not talking about a secure server (https), the best way is perhaps as Matt Cutts says in the video, to use .htaccess to password protect, that is authenticate the protected area for human beings. But if your content is open to one other person, it is potentially open to the world

[kgun]

Since this is also a security related topic, I started a new thread in the security sub forum:

Get IP with Socket & C++

[wige]

I just wanted to post this quick reply while I am thinking about it and before I read the rest of the intervening posts...

Originally Posted by Webnauts

If you are right, then the toolbar values are not just only out-of-date (2-4 months old), but they are faked!!!

This deserves a thread of its own!

<snip>

Well as I said if the noindex page has PageRank and it is not displayed in the toolbar, the toolbar values are fake. If that is they case, the toolbar sucks more than my imagination could perceive since I am working as an SEO professional.

Toolbar pagerank IS NOT internal pagerank. These are different values, scaled and calculated differently. They are based on the same information as internal pagerank, but they are very different values.

Internal pagerank is a value between 0 and 1, which is a ratio indicating the odds that a random surfer starting at a random page in the index and clicking a series of randomly selected links (3, if I recall correctly) will end up on the specified page.

Toolbar pagerank is a weighted scale of the relative link strength (number of inbound links weighted recursively by the number of inbound links to the linking pages) of the specified page, indicated on a scale from 1 to 10, weighted so that 1-2 contains exponentially more pages than 9-10.

To remember those pages do those pages need to have PageRank?

Technically, no. However, since a page can be marked as NOINDEX, FOLLOW and pass pagerank (and therefore must accumulate pagerank) it does not seem logical to me that a page marked as NOINDEX, NOFOLLOW would not accumulate pagerank simply because it has no followable links. Google's algorithm already has a mechanism to distribute the accumulated pagerank so I see no reason it would not be accumulated just because the links aren't followed.

[wige]

Originally Posted by Webnauts

Another question:

I have internal anchors on each page, i.e skip links for blind users, etc. like this:

<a href="#main-content" accesskey="2">Skip to main content</a>

Does Google assign Pagerank to those internal anchors?

No. By specification, browsers don't even send a request to the server for anchor links (on the same page). There is no reason to suspect that any search engine would take anchor links into account.

Next question:

I am on the services age of my site. For an appropriate navigation I need to have 3 links to my homepage.

1. In the logo.
2. In the main navigation.
3. In the breadcrumbs.

Now. If 3 links are pointing to the same page, won't I have a leak of PR?

That is all I would like to know, and then I am coming with more details.

Thanks.

This goes to the 1 page 1 vote concept. A page can only vote for another page once, so when the spider builds its list of pages that are linked to (and thus which should receive pagerank) duplicates are removed. Every page that is linked to receives the same portion of pagerank, regardless of how many actual links there are to the destination page.

However, because of this merging, there could be dilution of keywords. If you have two links with different link text, each keyword might give the target page less value.

[Terry Van Horne]

Originally Posted by wige

No. By specification, browsers don't even send a request to the server for anchor links (on the same page). There is no reason to suspect that any search engine would take anchor links into account.

It is however an attribute/part of the, IMO, second most important HTML element... the href (Title is King). Which could possibly be considered in the contextual link analysis (words the link is embedded in). That could be why FAQ pages sometimes do well. Not to say they do raise relevance or anything... I disagree there is no reason to suspect a search engine would be interested in the anchor.

[inertia]

A page can only vote for another page once, so when the spider builds its list of pages that are linked to (and thus which should receive pagerank) duplicates are removed. Every page that is linked to receives the same portion of pagerank, regardless of how many actual links there are to the destination page.

Can i ask for some clarification on this? Is this what you believe? Are you saying that if i have a menu link to a page but then also link to that page from within the body text then the second one isnt taken into consideration by G?

[kevsta]

im still trying to satisfy myself with a definitive answer to this by testing, inconclusively thus far..

[wige]

Originally Posted by inertia

Can i ask for some clarification on this? Is this what you believe? Are you saying that if i have a menu link to a page but then also link to that page from within the body text then the second one isnt taken into consideration by G?

Lets say you have a page with five links, each pointing to a page. That page of links has a pagerank of 1. (using the tbpr scale for simplicity here) As we know so far, Google will credit 1/5 of that 1 point of pr to each linked page. Simply put, Google doesn't give pass more or less pagerank if there is more than one link to a URL.

Now, lets say you edit this page, adding a second link to one of the pages you already linked to. Now you have six links to five URLs. For pagerank purposes, Google ignores (or more likely merges) the two links to the same URL and still credits each linked page with 1/5 of this page's 1 point.

As far as how keywords and text are handled, there are two possibilities. Google may only consider the first link, and discount the second. Personally, I don't think that is likely, but I don't have any way to be sure. I think, though, it is more likely that Google considers the context of every link to the destination URL, but weighs them less.

Imagine that the target page is about dogs and cats. The first link on the links page has the link text "dog", and that is the only link from the links page to the target. Google will consider that a 1/5 of 1 point vote that the target page is related to "dog". Now, lets say that you add the extra link, this time with the link text "cat". Some theorize that Google will only credit the target with the 1/5 of 1 point for the term "dog". I think that Google will split the difference, crediting the target with 1/10 of 1 point for "dog" and 1/10 of 1 point for "cat" - the target still gets 1/5 of 1 point of pagerank, but each individual keyword is worth less.

[inertia]

From a PR POV i understand what you mean completely. As far as the number of incoming internal links a page has... The basis of good site architecture is firing more internal links at the quality pages and the reason that Google tells us the number of internal links it sees for each page in GWT is to promote this sort of linking.

Also, from experience, going through a site and interlinking relevant keywords in the text of each page is a sure fire way to improve rankings, regardless of what other links are already hire up in the page code.

So i guess i think that each link to a page weighs things more as it's the extra links that are giving Google the clues to your sites architecture and each pages target keywords.

[crankydave]

Originally Posted by inertia

From a PR POV i understand what you mean completely. As far as the number of incoming internal links a page has... The basis of good site architecture is firing more internal links at the quality pages and the reason that Google tells us the number of internal links it sees for each page in GWT is to promote this sort of linking.

Also, from experience, going through a site and interlinking relevant keywords in the text of each page is a sure fire way to improve rankings, regardless of what other links are already hire up in the page code.

So i guess i think that each link to a page weighs things more as it's the extra links that are giving Google the clues to your sites architecture and each pages target keywords.

If what you're saying is that "normalizing" doesn't neccessary apply to all weighted aspects of a link then I agree.

Dave

[Webnauts]

Well I would like to get all this to a point keeping things as simple as possible.

So here is my questions:

1. What is the best robots.txt directive for blocking pages of being indexed by Google: Disallow or Noindex directive?

2. If external sites are linking to pages I do not want them to be indexed and receive PR, what is the best solution for that and why?

[inertia]

If what you're saying is that "normalizing" doesn't neccessary apply to all weighted aspects of a link then I agree.

Dave

Thats pretty much what i mean! The more you internally link to a page, the more important you are telling Google that it is. Unfortunately we tend to link to un-targeted pages more (contact, sitemap, login etc). So if you do that's where nofollow can be useful.

1. What is the best robots.txt directive for blocking pages of being indexed by Google: Disallow or Noindex directive?

Noindex? But arent you supposed to be answering that question for us?!

2. If external sites are linking to pages I do not want them to be indexed and receive PR, what is the best solution for that and why?

If they are duplicate content - 301. If they are pages that are poor quality (logins etc) pages then I'd just get the outbound links on them sorted so that the PR "flows" to the right places.

[kevsta]

Originally Posted by Webnauts

Well I would like to get all this to a point keeping things as simple as possible.

So here is my questions:

1. What is the best robots.txt directive for blocking pages of being indexed by Google: Disallow or Noindex directive?

2. If external sites are linking to pages I do not want them to be indexed and receive PR, what is the best solution for that and why?

John with respect you do seem unduly fixated, if not obsessed with robots.txt?

this is only one small part of a very large picture and many many sites absolutely rock in the charts without even using robots.txt

[kevsta]

Originally Posted by inertia

If they are pages that are poor quality (logins etc) pages then I'd just get the outbound links on them sorted so that the PR "flows" to the right places.

yea noindex disallow whatever if you have to, or just leave it.

I'd nofollow all the unimportant links (including nav if necessary) offpage and make sure a couple of nice anchors were left live to important pages.

[crankydave]

Originally Posted by Webnauts

1. What is the best robots.txt directive for blocking pages of being indexed by Google: Disallow or Noindex directive?

John... neither will accomplish what it is you are trying to do... Stop the page from accumulating PR AND send that incoming PR to a page of your choosing.

As I said earlier, if you wish to send the PR to a page of your choosing, use a redirect.

Originally Posted by Webnauts

2. If external sites are linking to pages I do not want them to be indexed and receive PR, what is the best solution for that and why?

Redirect the page John. But that's not what you want to accomplish.

You want to have a page. You want that page to get external links to it. You want to keep that page on your site. You want to send the PR from those external links directly to a different page(s) of your choosing. It simply doesn't work that way.

Actually, if you stop and think about it, if it were possible that would be just about as manipulative as you could possibly get.

Dave

[williamc]

Originally Posted by crankydave

Actually, if you stop and think about it, if it were possible that would be just about as manipulative as you could possibly get.

I would have to agree.

Heres the answer in a nutshell John: The only way to do what you want is IP Delivery or simply put, cloaking. If its a human visitor, show the page, if it is a search engine, or the toolbarquery, send a 301 redirect to send the PR elsewhere.

Manipulative as hell and bound to accrue a penalty when reviewed.

[Webnauts]

Originally Posted by inertia

Thats pretty much what i mean! The more you internally link to a page, the more important you are telling Google that it is. Unfortunately we tend to link to un-targeted pages more (contact, sitemap, login etc). So if you do that's where nofollow can be useful.

To Follow or to Nofollow… | SEO.com

Originally Posted by inertia

Noindex? But arent you supposed to be answering that question for us?!

I did but I feel that some members here disagree.

Originally Posted by inertia

If they are duplicate content - 301.

How? Thinks online shops.

Originally Posted by inertia

If they are pages that are poor quality (logins etc) pages then I'd just get the outbound links on them sorted so that the PR "flows" to the right places.

Again how? In example if you have an order form.

[Webnauts]

Originally Posted by kevsta

John with respect you do seem unduly fixated, if not obsessed with robots.txt?

I am not obsessed in any way. Looking for better professional solutions to herd the bots and Pagerank is obsession? Well if you think so, great for you brother.

Originally Posted by kevsta

this is only one small part of a very large picture and many many sites absolutely rock in the charts without even using robots.txt

A small part of a large picture?

I would like to share with you a new online shop launched end of February this year so you can tell me were it success comes from. If the mods don't mind if I will share the link here.

And by the way I was expecting a straightforward (even short) answer to my two questions above. So can you please...?

[Webnauts]

Originally Posted by crankydave

You want to have a page. You want that page to get external links to it. You want to keep that page on your site. You want to send the PR from those external links directly to a different page(s) of your choosing. It simply doesn't work that way.

Actually, if you stop and think about it, if it were possible that would be just about as manipulative as you could possibly get.

Dave

1. I do not want to get external links to link to it. Those are created without my permission.

2. Are you claiming that PR of pages being attributed with a "noindex" meta tag or robots.txt directive do not pass PR to other pages though the destination links placed that page?

If that is the case, thenwe have an entirely different opinion, and I am not willing to go into further discussions about facts that is already gathered.

Still I was expecting a short answer to questions 1 and 2. Short and straight to the point.

[Webnauts]

Originally Posted by williamc

I would have to agree.

Heres the answer in a nutshell John: The only way to do what you want is IP Delivery or simply put, cloaking. If its a human visitor, show the page, if it is a search engine, or the toolbarquery, send a 301 redirect to send the PR elsewhere.

Manipulative as hell and bound to accrue a penalty when reviewed.

Can you or anyone else answer this very simple question?

What is the best robots.txt directive for blocking pages of being indexed by Google: Disallow or Noindex directive?

(1) for Disallow
(2) for Noindex

Just that simple.

[Feydakin]

Which do you think is best??

[williamc]

Originally Posted by Webnauts

What is the best robots.txt directive for blocking pages of being indexed by Google: Disallow or Noindex directive?

The question you are asking needs to be clarified then, because first you are talking about bot herding, and pagerank transferance and then in the same thread ask how to stop a page from being indexed. Very different subjects.

Speak clearer.

But the direct question you posed above has only one answer. Disallow. Because NoIndex is not actually a robots.txt directive at all. It is a meta tag directive only.

[Webnauts]

Originally Posted by williamc

But the direct question you posed above has only one answer. Disallow. Because NoIndex is not actually a robots.txt directive at all. It is a meta tag directive only.

No no buddy. That is not right!

Originally Posted by williamc

The question you are asking needs to be clarified then, because first you are talking about bot herding, and pagerank transferance and then in the same thread ask how to stop a page from being indexed. Very different subjects.
Well I am talking all the time about these two different things.

Speak clearer.

Allow me to explain:

1. NoIndex is also a robots.txt directive supported unofficially by Google already since 2007. What year we have now? 2009? Does google support the 'Noindex' directive in the robots.txt file - Webmaster Help

2. Also check this excellent comment of Andy Beard: SEOmoz | Headsmacking Tip #13: Don't Accidentally Block Link Juice with Robots.txt
The best would be to read all comments of that post.

3. Noindex directive, no matter if you use it in a meta tag, x-robots or robots.txt has the same effect.

4. If you want to learn how the "noindex" is treated by Google, I think Matt Cutts can explain that for you: What should NOINDEX do?

5. At last, juice does not pass on from noindexed pages? http://eduardblacquiere.com/non-inde...pass-pagerank/

Back to SEO basics:

The "noindex" robots meta tag directive is for a single page and the "noindex" robots.txt directive is for a server level.

Are we clear now?

[williamc]

Actually, if you read the google forum post you so kindly provided, you will notice luzie saying it did not work and sagan including his remark in his post, saying he was of the ones correct. It is not formalised and there is no current evidence that shows it is still in use after the initial testing.

It is still formally only a meta directive. My answer remains unchanged and accurate.

I have always been clear

[Webnauts]

Originally Posted by williamc

Actually, if you read the google forum post you so kindly provided, you will notice luzie saying it did not work and sagan including his remark in his post, saying he was of the ones correct. It is not formalised and there is no current evidence that shows it is still in use after the initial testing.

I tested it already on many sites the last 11 months.

Go in your Google Webmaster Tools and test this robots.txt, and when you do that please come back and tell us what happened.
User-agent: Googlebot
Noindex: /*?
Noindex: *bots=nocrawl
Noindex: /*info_
Noindex: /*tellafriend
Noindex: /images/upload
Noindex: /*cat_
Noindex: /*prod_
Allow: /contact/info_2.html$
Allow: /sitemap/info_5.html$
Allow: /*cat_*.html$
Allow: /*prod_*.html$

Are you also saying that my robots.txt don't work? http://www.seoworkers.com/robots.txt

If that is what you believe, I have some evidence that I can show off.

Originally Posted by williamc

It is still formally only a meta directive. My answer remains unchanged and accurate.

I have always been clear

Well John Mu from Google seems to agree with me. http://www.pagetrafficblog.com/googl...ermanent/3523/ LOL

[Webnauts]

And for everybody else here:

1. Interview with Google's Adam Lasnik - Web Design & SEO Company

2. http://sebastians-pamphlets.com/abou...in-robots-txt/

3. http://sebastians-pamphlets.com/vali...dex-your-site/

[Webnauts]

I am happy that I could find the time to continue this discussion. Now I came up with an idea.

I can setup four pages on my web site.

The page A will have a meta robots "noindex and the page B will have the "noindex" robots.txt directive.

Then on page A I will add a link to the page C (not blocked with a noindex meta or robots.txt directive), and on page B I will add a link to page D also not be blocked with a noindex meta or robots.txt directive.

Then I will get backlinks for the pages A and B. If you want you can help there.

Lets site back and watch what will happen. I did that test already, but we can repeated again. Or does someone else what to do the test. I would be glad to contribute.

[williamc]

I take it that you did not actually read all of the post which had John Muller's comment.

It says that google will "accept" reading the noindex in the robots.txt, and that it would not "break" things.

That was the thrust of the article, of which they used Johns comment as backup, it did not say that it was anything but a one time experiment and would not be shut off, if it has not already. It simply said that it would not break googles parsing of the robots.txt file.

Secondly, simply because using it in robots.txt does not "break" anything, does not mean they will actually pay it the same attention as the noindex meta directive, or being an experiment, that they will continue to do so.

Next let's look at sebastians pamphlets of which I was already aware and hoping you might use. In there we have this "hint":

A: The “noindex” value is a crawler directive valid in robots meta tags as well as X-Robots-Tags, not (yet) in robots.txt. Maybe Google has something cooking, or the “Noindex: /” statement is just an experiment.

Notice the not (yet) in robots.txt bit...

Now corelating that with what the google forum answer you originally posted gives us "an experiment" several times.Other experimental robots.txt directives are nosnippet and noindex-follow. These experiments can or could have ended by now, and when they do, at some point can break the validation of your robots.txt

Now on to your adam lasnik article: Adam says, and I quote

So, my core suggestion to webmasters would be to use Noindex and robots.txt to help us know what pages you'd prefer to not have indexed

Notice he does not say use NoIndex in robots.txt, but instead says NoIndex and robots.txt and then goes on to give some examples for using robots.txt to block content and rss feeds, but does not ever say to use NoIndex in it to do so as disallow will stop that content from being indexed.

As I already said: It is not a formal directive. My answer stands.

[Webnauts]

Here is some an update to the above links, posts, comments, etc.

Matt Cutts said:

Google allows a NOINDEX directive in robots.txt and it will completely remove all matching site urls from Google.

Source: What should NOINDEX do?

[Webnauts]

Originally Posted by wige

To kind of restate what John has said above, and to make sure that I am properly understanding the way that pagerank works with current technologies, lets imagine two pages. One page has a link on it (we will call this the source page) and the other page is the page that the link points to (destination page). The flow of pagerank is calculated when the source page is crawled, and is not affected by any factors other than those on the source page; ie, the flow of pagerank can be affected by a nofollow attribute on the link itself, and the flow of pagerank can be affected if the source page is blocked by a disallow directive because Google can't see the links. Google does not consider anything about the destination page when calculating the pagerank a destination page receives.

EDIT to add: A page that has a link always recieves pagerank. A link that is marked as nofollow (either with a meta tag or a rel attribute) is the only exception. A page that is marked as noindex will still recieve pagerank, that pagerank simply can't be seen (since the page doesn't get put into the index, you have no way of asking Google what the pagerank of the page is - Google still knows that the url was linked to, and pagerank is still allocated for that url because the concept that the page is set to noindex is unknown to Google when the pagerank from the source page is being divided amonst the outgoing links.)

There are a few different cases that you can encounter a "dead end". The original "dead ends" were pages that had incoming links, but no outgoing links. Pages that have all of their outgoing links marked with nofollow would fall into the same category (nofollow meta tag or all links with a nofollow attribute), as would pages that can't be crawled (since Google can't see the links on the page) due to a disallow robots.txt directive, or pages that go through a 302 redirect (Google does this intentionally, so that 302 redirects can't pass pagerank).

What Google does with a dead end is they take all the pagerank from all the known dead ends in the index, and then distribute that pagerank amongst all the other pages in the entire index. As a result, a dead end provides little to no actual benefit to the website itself. In fact, it can harm the web site by draining some of the site's pagerank which could be better spent pointing to more important pages. As a result, you would want to keep the pagerank flowing anywhere but to a dead end.

However, there is no way to automatically stop pagerank from going to deadends - you can only change the flow of pagerank on the source page, never the destination page. The only option is to manually add a nofollow attribute to every link that goes to a dead end - not an ideal option.

So, what I figured out while typing this overly long summary, is that 403 pages provide a way to recover pagerank that would otherwise go to a dead end. Basically, the trick is in the error message. What you are doing is giving users a way (through authentication) to see the content that you don't want the spiders to crawl, while giving the spiders an error page that has links that recovers the pagerank and sends it to the important areas of your site.

Wige shall we continue from here?

As I posted above, Matt Cutts said:

Google allows a NOINDEX directive in robots.txt and it will completely remove all matching site urls from Google.

And... I was just reading:

Dropping URLs - The URL scheduler determines which URLs are deleted from data structure and therefore dropped from system. URLs might be removed to make room for new URLs. A “keep score” might be computed for each URL, and the URLs could then be sorted by the “keep score.”
URLs with a low “keep score” could be eliminated as newly discovered URLs are added. The “keep score” could be the page rank of a URL determined by page rankers.

Source: Google Patent on Anchor Text and Different Crawling Rates

So what will happen if I tell Google not to index a page implementing the "noindex" meta tag or robots.txt directive?

Looking very much forward for your thoughts.

[crankydave]

Originally Posted by Webnauts

1. I do not want to get external links to link to it. Those are created without my permission.

But you want to attribute the the PR you are receiving from those external links to pages of your choosing. Redirect the page or as Will pointed out, cloak it.

Created without your permission? Then noindex nofollow the page. The page won't be displayed in the results and won't distribute PR to any other pages it links to.

But that's not what you want to do is it?

Originally Posted by Webnauts

2. Are you claiming that PR of pages being attributed with a "noindex" meta tag or robots.txt directive do not pass PR to other pages though the destination links placed that page?

Did you read my post John? That's not even close to what I said. I said this on page 1...

Naturally the toolbar wouldn't show PR. That doesn't mean the PR coming from links "jumps" to pages it links to. The noindex page accumulates PR it's simply not "seen" via the toolbar. A noindex page accumulates and passes PR in the same way an indexed page would.

Originally Posted by Webnauts

Still I was expecting a short answer to questions 1 and 2. Short and straight to the point.

Question 1: A disallowed page can show up as a URL only listing. A noindex page won't. But you already know this.

Question 2: You can't prevent a page from receiving PR. You can redirect that page, you can remove it altogether, or you can cloak it. You already know this too.

Dave

[Webnauts]

Originally Posted by crankydave

But you want to attribute the the PR you are receiving from those external links to pages of your choosing. Redirect the page or as Will pointed out, cloak it.

To be honest I still don't trust Google enough.

Originally Posted by crankydave

Created without your permission? Then noindex nofollow the page. The page won't be displayed in the results and won't distribute PR to any other pages it links to.

But that's not what you want to do is it?

No, that is not what I want to do. What I want to do is:

<meta name="robots" content="noindex,follow" />

or

User-agent: Googlebot
Noindex: / donotindexpagename.html

Originally Posted by crankydave

Did you read my post John? That's not even close to what I said. I said this on page 1...

Sorry Dave. Then I misunderstood.

Originally Posted by crankydave

Question 1: A disallowed page can show up as a URL only listing. A noindex page won't. But you already know this.

That is what I am preaching for at least 2 years.

Originally Posted by crankydave

Question 2: You can't prevent a page from receiving PR. You can redirect that page, you can remove it altogether, or you can cloak it. You already know this too.

Dave

Matt Cutts said:

Google allows a NOINDEX directive in robots.txt and it will completely remove all matching site urls from Google.

Patent of Google:

Dropping URLs - The URL scheduler determines which URLs are deleted from data structure and therefore dropped from system. URLs might be removed to make room for new URLs. A “keep score” might be computed for each URL, and the URLs could then be sorted by the “keep score.”
URLs with a low “keep score” could be eliminated as newly discovered URLs are added. The “keep score” could be the page rank of a URL determined by page rankers.

At last, please do me a favor and read this article SEO Question: does a non-indexed page pass on PageRank? - Eduard Blacquière’s Search Marketing Blog : Eduard Blacquière’s Search Marketing Blog and lets can go on from there.

Thanks man.

[crankydave]

Originally Posted by Webnauts

No, that is not what I want to do. What I want to do is:

<meta name="robots" content="noindex,follow" />

or

User-agent: Googlebot
Noindex: / donotindexpagename.html

Then do it. There's nothing stopping you. What's the problem exactly?

Originally Posted by Webnauts

At last, please do me a favor and read this article...

There's nothing in that article that hasn't already been said in this thread. What was the point of me reading it?

Dave

[Webnauts]

Originally Posted by crankydave

Then do it. There's nothing stopping you. What's the problem exactly?



There's nothing in that article that hasn't already been said in this thread. What was the point of me reading it?

Dave

I mentoned above the quote of Matt Cutts, that Google allows a NOINDEX directive in robots.txt and it will completely remove all matching site urls from Google.

Google patent say's something about URLs with low "keep score":

The URL scheduler determines which URLs are deleted from data structure and therefore dropped from system. URLs might be removed to make room for new URLs. A “keep score” might be computed for each URL, and the URLs could then be sorted by the “keep score.”
URLs with a low “keep score” could be eliminated as newly discovered URLs are added. The “keep score” could be the page rank of a URL determined by page rankers.

My questions are:

1. If a page is blocked of being indexed with meta tag or robots.txt, and all matching URLs are completely removed from Google, can that page have PR for itself?

2. If that is not the case, could it be that those pages URL have a low "keep score"? If yes, what happens when I add new pages. Could it be that those URLs will go to Nirvana? At Nirvana there is no PageRank. Don't you agree?

[crankydave]

Originally Posted by Webnauts

1. If a page is blocked of being indexed with meta tag or robots.txt, and all matching URLs are completely removed from Google, can that page have PR for itself?

<sigh>

Yes John. A noindex page can and does accumulate PR. MC has stated as much. A page cannot pass PR if it doesn't have PR.

Originally Posted by Webnauts

2. If that is not the case, could it be that those pages URL have a low "keep score"? If yes, what happens when I add new pages. Could it be that those URLs will go to Nirvana? At Nirvana there is no PageRank. Don't you agree?

<sigh again>

No we do not agree. A page that has a link pointing to it, one that is followed, must have PR once that link is found.

Dave

[williamc]

Originally Posted by Webnauts

Here is some an update to the above links, posts, comments, etc.

Matt Cutts said:



Source: What should NOINDEX do?

Yes John, we already ascertained that it exists/existed, as an "experiment" that should "not be relied upon" and "could disapear at any time". You miss the point entirely. It is not a formal directive and not reliable. In fact one of the above quoted sources even went so far as to say google "will usually" recognize it during the experiment.

Usually... Not always...

As stated my answer stands. The only formal directive is disallow.

You can keep arguing the point til the cows come home, but so far every source you have shown, besides an offhand remark in passing by Matt, back up my version and not your own.

[williamc]

Originally Posted by crankydave

No we do not agree. A page that has a link pointing to it, one that is followed, must have PR once that link is found.

hear hear, run it home brother!

[Webnauts]

Ok, ok, ok.

Before the cow comes home, I thought of asking my last questions and then I can go in peace.

First lets look at the interview of Eric Enge with Matt Cutts:

Eric Enge Can a NoIndex page accumulate PageRank?

Matt Cutts A NoIndex page can accumulate PageRank, because the links are still followed outwards from a NoIndex page.

Eric Enge So, it can accumulate and pass PageRank.

Matt Cutts Right, and it will still accumulate PageRank, but it won’t be showing in our Index. So, I wouldn’t make a NoIndex page that itself is a dead end. You can make a NoIndex page that has links to lots of other pages.

Eric Enge For example you might want to have a master Sitemap page and for whatever reason NoIndex that, but then have links to all your sub Sitemaps.
…

Eric Enge Another example is if you have pages on a site with content that from a user point of view you recognize that it’s valuable to have the page, but you feel that is too duplicative of content on another page on the site

That page might still get links, but you don’t want it in the Index and you want the crawler to follow the paths into the rest of the site.

Matt Cutts That’s right. Another good example is, maybe you have a login page, and everybody ends up linking to that login page. That provides very little content value, so you could NoIndex that page, but then the outgoing links would still have PageRank.
Now, if you want to you can also add a NoFollow metatag, and that will say don’t show this page at all in Google’s Index, and don’t follow any outgoing links, and no PageRank flows from that page. We really think of these things as trying to provide as many opportunities as possible to sculpt where you want your PageRank to flow, or where you want Googlebot to spend more time and attention

Now lets leave the "noindex" robots.txt directive out of here, and lets move and stick on the robots meta tag directives "noindex" and "nofollow", and to the robots.txt directive "disallow". OK?

I have the following scenario:

I have a login page, and everybody ends up linking to a login page. And that provides very little content value. So which path should I follow?

Add to that login page:

a. <meta name="robots" content="noindex" />

or

b. Robots.txt "disallow" directive?

Once you choose between a or b, can you please explain why you have chosen that.

Thanks.

[williamc]

In those circumstances, at least the noindex allwos you to pass the accumalated PR to other pages in your site. If it is a login page, you could easily not show any links to outside sites, and only show a certain set of interior outbound links to pass or scult the PR where you want it flowing.

[Webnauts]

Originally Posted by williamc

In those circumstances, at least the noindex allwos you to pass the accumalated PR to other pages in your site. If it is a login page, you could easily not show any links to outside sites, and only show a certain set of interior outbound links to pass or scult the PR where you want it flowing.

Great. And if one day the "noindex" robots.txt directive will be officially supported by Google, the we can use that for individual pages or whole folders. Right?

And what about using the X-Robots "noindex" directive?

[williamc]

Originally Posted by Webnauts

Great. And if one day the "noindex" robots.txt directive will be officially supported by Google, the we can use that for individual pages or whole folders. Right?

And what about using the X-Robots "noindex" directive?

Re: robots.txt and noindex: definately. I just feel that using it now when they have stated it may break things down the road is a bit silly.

X-Robots is parsed exactly the same as robots as far as I am aware. I could be wrong on that one tho.

[inertia]

If everyone ends up linking to your login page why dont you... 301 it to somewhere else and add a site wide login box to the design?!

[williamc]

Originally Posted by inertia

If everyone ends up linking to your login page why dont you... 301 it to somewhere else and add a site wide login box to the design?!

Now see, there is a workable solution.

[crankydave]

I would do neither John.

If people want to link to that page, let them. Let people find it so they can continue to link to it. Obviously, they find it important enough to link to. Why stop them?

Provide a snippet of relevant text and some links to your pages with that relevant information and go about your business.

Or...

Create a new login page and 301 the old one to where ever you like. Or as Matt pointed out above, a sitewide login box is a very good suggestion.

Dave

[Feydakin]

Uh, John, aren't we supposed to be building our web sites for the user and not the search engines?? Your users obviously want to link to, and get to, your log in page, why would you try to game the search engines to take advantage of that??

[Webnauts]

Originally Posted by Feydakin

Uh, John, aren't we supposed to be building our web sites for the user and not the search engines?? Your users obviously want to link to, and get to, your log in page, why would you try to game the search engines to take advantage of that??

Who said they should not link to the login page? Who said they should not have access the login page? Are you probably not reading the posts here?