The Power of the WebProWorld SEO Community

[Webnauts]

You should already know that here at WPW there are high competent SEO professionals, a lot of beginneror intermediate level knowledge members.

What makes WPW a powerfull SEO Community? If all members no matter of which knowledge level will unite, I can't believe that exists out there an SEO professional how can compete us.

What I am after? Well ... optimize WPW? Just kidding. Well... if they need help, we sure can give a hand.

Before I come to my point, first I would like to make something clear:

I do not want any "thank you for great information", or any of those filthy and annoying posts of members who are trying to draw the attention of the thread subscribers to their signature, selling laptop batteries, or what ever else...

If my condition will be violated, and the mods will not take action, I will leave the thread imediatelly.

Now here I want to bring up a very interesting topic, that everyone will be thankful with the come out. So please don't go off-topic.
Please take some minutes and watch the video below:

YouTube - Matt Cutts Discusses Webmaster Tools

Notice: I will add the link of this thread at Youtube as a comment inviting Matt to come over if he has time.

So I have two points I would love to ask Matt.

1. He said that one of the best solutions (with a smiley) is to protect areas via .htaccess password protected area.

My concerns:

The IBM patent explores dangling nodes in depth, and provides a list of pages that may be treated by search engines as dangling nodes. And one of them is:

- If the page requires authentication.

What happens when a without a nofollow attribute or masked client/server side link is pointing to the protected area directory?

2. He mentioned the use of the noindex meta tag, but that is not the best solution. I am affraid that I have to disagree there. I will explain later on why.

My concerns:

a. Why didn't he mention the noindex robots.txt directive? Because it is unofficially supported?

b. If I disallow through my robots.txt a protected area with a htpasswd via .htaccess, I have a dangling/node issue, or?

Wouldn't noindex via robots.txt be better alternative? I am still thinking about it...

What's next?

Shoot your thoughts!

[inertia]

I guess what we really need to know is Matts/Google official stance on the nofollow robots.txt directive! If google could confirm how they handle this all this debate would be resolved!

[kgun]

John, I have other concerns:

What about webmasters using Microsoft's IIS web server? About 30 % market share april 2009

Links:

Easiest way to get .htaccess like access control with IIS 6? | Ask Metafilter

HOW TO: Migrate .Htaccess Data in a UNIX-to-Windows Migration

Personally I combine robots.txt (disallow file types and folders) and .htaccess (deny access to a folder and sub folders).

This discussion started here (post #159)

New Canonical Tag from the big 3

[wige]

Well, in principle I would agree that if you don't want spiders to access certain content, the best way to do so is to use authentication (to jump ahead to another point, I believe this is fairly easy to do in IIS also, it is just done with file-system permissions rather than through the .htaccess file itself). The noindex directive is unofficial, and support seems to vary. A page that is password protected is generally safe against all bots, even spambots that ignore robots.txt directives. This is particularly the case in some of the prime examples used to justify the creation of the robots.txt system - voting systems, and other pages where non-user access would cause problems.

That being said, I am not sure that the IBM patent would have any effect. There is no evidence that this patent has been picked up by any of the major search engines. It seems to me that the patent is a proposed modification to Google's existing pagerank model, and I am not sure that Google would modify that model, which so far has served them quite well. But, if anything, the current known Google model for distributing pagerank would be more adversely affected. Right now, pagerank is distributed among all outgoing links, regardless of destination, unless the link is marked as nofollow. This means that even if the link points to somewhere that Google can't crawl/index, the link still gets pagerank, which means that a 403 destination could become a pagerank black hole.

In IBM's patent, pagerank would be distributed only if the destination is crawlable, and meets certain other requirements. If you want to think of it in Google terms, the search engine could decide to apply a rel=nofollow attribute to a link on it's own, removing links from the pagerank calculations based on information about the crawlability, doc type, robots.txt status, etc. of the destination.

As a result of this, it is under the current model that a plain link to the authenticated content would be a potential problem. And, as mentioned before, I am not convinced that the IBM model is actually in use anywhere right now.

[Webnauts]

Originally Posted by inertia

I guess what we really need to know is Matts/Google official stance on the nofollow robots.txt directive! If google could confirm how they handle this all this debate would be resolved!

I personally do not need to ask Matt/Google official stance on the noindex robots.txt directive. By the way nofollow robots.txt does not exist.

I do not need to ask anybody since I already implemented on all my sites and on all sites of my customers and works as it should. I hope that is fair enough, or?

[Webnauts]

Originally Posted by kgun

Personally I combine robots.txt (disallow file types and folders) and .htaccess (deny access to a folder and sub folders).

Using .htaccess to deny to a folder and to sub folder is a bad thing to do. I mean if you are returning a 403. That is because your are creating a dangling/node page, which are otherwise called dead end pages/folders.

[Webnauts]

Originally Posted by wige

Well, in principle I would agree that if you don't want spiders to access certain content, the best way to do so is to use authentication (to jump ahead to another point, I believe this is fairly easy to do in IIS also, it is just done with file-system permissions rather than through the .htaccess file itself). The noindex directive is unofficial, and support seems to vary. A page that is password protected is generally safe against all bots, even spambots that ignore robots.txt directives. This is particularly the case in some of the prime examples used to justify the creation of the robots.txt system - voting systems, and other pages where non-user access would cause problems.

That being said, I am not sure that the IBM patent would have any effect. There is no evidence that this patent has been picked up by any of the major search engines. It seems to me that the patent is a proposed modification to Google's existing pagerank model, and I am not sure that Google would modify that model, which so far has served them quite well. But, if anything, the current known Google model for distributing pagerank would be more adversely affected. Right now, pagerank is distributed among all outgoing links, regardless of destination, unless the link is marked as nofollow. This means that even if the link points to somewhere that Google can't crawl/index, the link still gets pagerank, which means that a 403 destination could become a pagerank black hole.

In IBM's patent, pagerank would be distributed only if the destination is crawlable, and meets certain other requirements. If you want to think of it in Google terms, the search engine could decide to apply a rel=nofollow attribute to a link on it's own, removing links from the pagerank calculations based on information about the crawlability, doc type, robots.txt status, etc. of the destination.

As a result of this, it is under the current model that a plain link to the authenticated content would be a potential problem. And, as mentioned before, I am not convinced that the IBM model is actually in use anywhere right now.

Excellent post! About the 403 destination, that is what I met in my response at the post of Kgun above.

About how Google handles the noindex directive, I am not waiting for Google to tell me how they handle it, since as I said above I am using is on many sites and it is very effective. It really does the same job as the noindex meta robots tag on a server level.

About IBMs patent if it is accurate or not, or if Google uses that or not, I do not really care, because I implement the necessary stuff for the case it does exist or will come exist in the future.

[inertia]

I personally do not need to ask Matt/Google official stance on the noindex robots.txt directive. By the way nofollow robots.txt does not exist.

I do not need to ask anybody since I already implemented on all my sites and on all sites of my customers and works as it should. I hope that is fair enough, or?

I meant noindex robots.txt directive. It just seems odd that Google or Matt C never mention it? Why would that be the case if it was such a useful tool that would benefit everyone including Google?

It really does the same job as the noindex meta robots tag on a server level.

Can i just clarify... If i block a page in robots.txt with the NOINDEX directive will that stop the page being crawled, indexed and also stop it building or leaking pagerank?

[Webnauts]

Originally Posted by inertia

I meant noindex robots.txt directive. It just seems odd that Google or Matt C never mention it? Why would that be the case if it was such a useful tool that would benefit everyone including Google?

I guess that Matt does not praise it because it is still unofficially supported. If I do not recall, Adam Lasnik talked about it a while ago. If I find the link I will post it here.

Originally Posted by inertia

Can i just clarify... If i block a page in robots.txt with the NOINDEX directive will that stop the page being crawled, indexed and also stop it building or leaking pagerank?

If you block a page in the robots.txt with the noindex directive it will not stop Googlebot crawling the pages, but Google will not index it.

What is the difference with the disallow directive?

In case someone is linking to a page you block with the disallow directive in the robots.txt, Googlebot will return the reference in their in their index but without a snippet. And there you have a PR leak, since PR will be assigned to that page too.

If you block that page with the noindex directive, the reference will not show up at all. And PR will not be assigned to that page, but still it will pass to other pages you link from there. But that to happen, you must have on that page at least one outbound (internal or external) link without a nofollow or so ever, so the PR can move ahead.

If not, then you will create dangling/nodes (dead end or hanging pages), in other words as Wige said above a PR black-hole.

[inertia]

If you block that page with the noindex directive, the reference will not show up at all. And PR will not be assigned to that page,

OK. I understand all the theory but one thing i dont understand regards the internal flow of pagerank... Do pages blocked with the robots noindex directive still build PR from internal links? If they dont then i can stop using nofollow tags! But as you've just started using them i guess they dont?

[Webnauts]

Originally Posted by inertia

OK. I understand all the theory but one thing i dont understand regards the internal flow of pagerank...

First it is not a theory. It is a evidence-based research fact.

Originally Posted by inertia

Do pages blocked with the robots noindex directive still build PR from internal links?

No.

Originally Posted by inertia

If they dont then i can stop using nofollow tags! But as you've just started using them i guess they dont?

Yes. But again, if someone else is linking to those pages they will build PR to them. And there is were you leak PR. Do you want pages you use the nofollow to have PR? Or do you prefer that the PR would pass to pages of your choice?

[inertia]

Originally Posted by Webnauts

First it is not a theory. It is a evidence-based research fact.

Just a confusion in my English there john. I meant theory like music theory or mathematical theory but saying that it is still theoretical to me as i havent tried it out myself yet!

Originally Posted by Webnauts

No.

But this is whats confusing me.... You're saying that a noindex page can still build and pass PR from outside links. Matt C also says that a noindex page with meta noindex can still build PR so isnt this just a difference in granularity?

Originally Posted by Webnauts

Yes. But again, if someone else is linking to those pages they will build PR to them. And there is were you leak PR. Do you want pages you use the nofollow to have PR? Or do you prefer that the PR would pass to pages of your choice?

You've lost me... Does the noindex robots.txt directive basically mean that i have added nofollow attributes to all the links pointing to the blocked page? Forget about external links for a minute... we've resolved that issue. Lets think about internal pr scuplting here!

[wige]

To kind of restate what John has said above, and to make sure that I am properly understanding the way that pagerank works with current technologies, lets imagine two pages. One page has a link on it (we will call this the source page) and the other page is the page that the link points to (destination page). The flow of pagerank is calculated when the source page is crawled, and is not affected by any factors other than those on the source page; ie, the flow of pagerank can be affected by a nofollow attribute on the link itself, and the flow of pagerank can be affected if the source page is blocked by a disallow directive because Google can't see the links. Google does not consider anything about the destination page when calculating the pagerank a destination page receives.

EDIT to add: A page that has a link always recieves pagerank. A link that is marked as nofollow (either with a meta tag or a rel attribute) is the only exception. A page that is marked as noindex will still recieve pagerank, that pagerank simply can't be seen (since the page doesn't get put into the index, you have no way of asking Google what the pagerank of the page is - Google still knows that the url was linked to, and pagerank is still allocated for that url because the concept that the page is set to noindex is unknown to Google when the pagerank from the source page is being divided amonst the outgoing links.)

There are a few different cases that you can encounter a "dead end". The original "dead ends" were pages that had incoming links, but no outgoing links. Pages that have all of their outgoing links marked with nofollow would fall into the same category (nofollow meta tag or all links with a nofollow attribute), as would pages that can't be crawled (since Google can't see the links on the page) due to a disallow robots.txt directive, or pages that go through a 302 redirect (Google does this intentionally, so that 302 redirects can't pass pagerank).

What Google does with a dead end is they take all the pagerank from all the known dead ends in the index, and then distribute that pagerank amongst all the other pages in the entire index. As a result, a dead end provides little to no actual benefit to the website itself. In fact, it can harm the web site by draining some of the site's pagerank which could be better spent pointing to more important pages. As a result, you would want to keep the pagerank flowing anywhere but to a dead end.

However, there is no way to automatically stop pagerank from going to deadends - you can only change the flow of pagerank on the source page, never the destination page. The only option is to manually add a nofollow attribute to every link that goes to a dead end - not an ideal option.

So, what I figured out while typing this overly long summary, is that 403 pages provide a way to recover pagerank that would otherwise go to a dead end. Basically, the trick is in the error message. What you are doing is giving users a way (through authentication) to see the content that you don't want the spiders to crawl, while giving the spiders an error page that has links that recovers the pagerank and sends it to the important areas of your site.

[Webnauts]

Originally Posted by inertia

Just a confusion in my English there john. I meant theory like music theory or mathematical theory but saying that it is still theoretical to me as i havent tried it out myself yet!

Got ya! I was just teasing you. But I did sound being serious...

Originally Posted by inertia

But this is whats confusing me.... You're saying that a noindex page can still build and pass PR from outside links. Matt C also says that a noindex page with meta noindex can still build PR so isnt this just a difference in granularity?

OK. Lets take an example:

Page A has PR 4. On page A there is a link to page B without being attributed by the nofollow attribute or so ever. Right?

Now, on page B there is a link to page C. OK?

What happens? Googlebot will follow the link on page A to page B but it will see that the page should not be taken into account, but still it will crawl the page looking for a link to pass the PR to. If there is no link or if the link(s) are attributed by a nofollow attribute or otherwise blocked, you will have a dead end (dangling/nodes).

Originally Posted by inertia

You've lost me... Does the noindex robots.txt directive basically mean that i have added nofollow attributes to all the links pointing to the blocked page?

No. That happens only if you would use the nofollow meta tag.

Originally Posted by inertia

Forget about external links for a minute... we've resolved that issue. Lets think about internal pr scuplting here!

Sure. We are only talking about internal PR sculpting.

[kgun]

Originally Posted by Webnauts

Using .htaccess to deny to a folder and to sub folder is a bad thing to do. I mean if you are returning a 403. That is because your are creating a dangling/node page, which are otherwise called dead end pages/folders.

Is this wrong http://www.kjellbleivik.com/Books/

.htaccess

order deny,allow
deny from all

except white listed IP's in the /Books/ folder ?

1. Will the white list create problems?
2. More precisely: Can Bot's get an URI reference or a dangling node (link) via the white list?

[crankydave]

Actually, should be rather easy to test.

Build and orphan a few pages. Four should be plenty... A,B,C,D

Link A to B... B to C... C to D

Noindex A, B, and C

Throw a single link from a page with a high toolbar PR at A. If the PR "jumps" all the way to D then it will display a high toolbar PR as well. If it passes through all the pages then it won't. Also, remove the noindex and see if there is any change.

Whether or not a page is noindex really doesn't matter. If there is an external link pointing to it then there is a "probability" that it can be reached by a random surfer. Ergo, it "has" PR once Google finds the link.

The more links a random surfer has to follow to ultimately get to the "destination" page, the less probability it will be reached... less PR.

Dave

[kgun]

Originally Posted by Webnauts

First it is not a theory. It is a evidence-based research fact.

Is that possible, research without a theory?

[wige]

Originally Posted by kgun

Is this wrong http://www.kjellbleivik.com/Books/

.htaccess

order deny,allow
deny from all

except white listed IP's in the /Books/ folder ?

1. Will the white list create problems?
2. More precisely: Can Bot's get an URI reference or a dangling node (link) via the white list?

If you whitelist the search engines, then yes, you will be fine from a pagerank standpoint. However, if the bots just get the default error message that non-whitelisted users get, then the page is a dead end. The custom error document is not being displayed, so there are no links to recieve the pagerank that is going to this page.

[Webnauts]

Originally Posted by wige

To kind of restate what John has said above, and to make sure that I am properly understanding the way that pagerank works with current technologies, lets imagine two pages. One page has a link on it (we will call this the source page) and the other page is the page that the link points to (destination page). The flow of pagerank is calculated when the source page is crawled, and is not affected by any factors other than those on the source page; ie, the flow of pagerank can be affected by a nofollow attribute on the link itself, and the flow of pagerank can be affected if the source page is blocked by a disallow directive because Google can't see the links.

If I understood correct, we agree.

Originally Posted by wige

Google does not consider anything about the destination page when calculating the pagerank a destination page receives.

Exactly.

Originally Posted by wige

EDIT to add: A page that has a link always recieves pagerank. A link that is marked as nofollow (either with a meta tag or a rel attribute) is the only exception. A page that is marked as noindex will still recieve pagerank, that pagerank simply can't be seen (since the page doesn't get put into the index, you have no way of asking Google what the pagerank of the page is

I think I must disagree at sme point. A page that is not indexed can not have PR its self. But the PR coming to the noindex attributed page will flow to the pages linked from it.

Originally Posted by wige

Google still knows that the url was linked to, and pagerank is still allocated for that url because the concept that the page is set to noindex is unknown to Google when the pagerank from the source page is being divided amonst the outgoing links.)

I am not sure if I understand.

Originally Posted by wige

There are a few different cases that you can encounter a "dead end". The original "dead ends" were pages that had incoming links, but no outgoing links.
Pages that have all of their outgoing links marked with nofollow would fall into the same category (nofollow meta tag or all links with a nofollow attribute), as would pages that can't be crawled (since Google can't see the links on the page) due to a disallow robots.txt directive, or pages that go through a 302 redirect (Google does this intentionally, so that 302 redirects can't pass pagerank).

Correct.

Originally Posted by wige

What Google does with a dead end is they take all the pagerank from all the known dead ends in the index, and then distribute that pagerank amongst all the other pages in the entire index. As a result, a dead end provides little to no actual benefit to the website itself. In fact, it can harm the web site by draining some of the site's pagerank which could be better spent pointing to more important pages. As a result, you would want to keep the pagerank flowing anywhere but to a dead end.

Exactly.

Originally Posted by wige

However, there is no way to automatically stop pagerank from going to deadends - you can only change the flow of pagerank on the source page, never the destination page. The only option is to manually add a nofollow attribute to every link that goes to a dead end - not an ideal option.

I agree.

Originally Posted by wige

So, what I figured out while typing this overly long summary, is that 403 pages provide a way to recover pagerank that would otherwise go to a dead end. Basically, the trick is in the error message. What you are doing is giving users a way (through authentication) to see the content that you don't want the spiders to crawl, while giving the spiders an error page that has links that recovers the pagerank and sends it to the important areas of your site.

You mean creating a custom 403 template with links to pages I want Googlebot to crawl and pass the incoming PR? I am not sure how can that work. Or am I confused?

[kgun]

Originally Posted by wige

However, if the bots just get the default error message that non-whitelisted users get, then the page is a dead end. The custom error document is not being displayed, so there are no links to recieve the pagerank that is going to this page.

And your solution (redirect) is?

What were discussed in these two threads:

403 Forbidden error document::: How to do it?

Custom error pages::: Best practices.

[Webnauts]

Originally Posted by kgun

Is this wrong http://www.kjellbleivik.com/Books/

I saw there an alternative 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request. But doing an HTTP Header check I was getting only the 403. So doesn't look good to me.

Originally Posted by kgun

.htaccess

order deny,allow
deny from all

except white listed IP's in the /Books/ folder ?

1. Will the white list create problems?
2. More precisely: Can Bot's get an URI reference or a dangling node (link) via the white list?

Are the bots also white listed? I don't understand.

[crankydave]

Originally Posted by Webnauts

I think I must disagree at sme point. A page that is not indexed can not have PR its self. But the PR coming to the noindex attributed page will flow to the pages linked from it.

Eric Enge Can a NoIndex page accumulate PageRank?

Matt Cutts A NoIndex page can accumulate PageRank, because the links are still followed outwards from a NoIndex page.

Eric Enge So, it can accumulate and pass PageRank.

Matt Cutts Right, and it will still accumulate PageRank, but it won’t be showing in our Index. So, I wouldn’t make a NoIndex page that itself is a dead end. You can make a NoIndex page that has links to lots of other pages.
Eric Enge Interviews Google’s Matt Cutts

Control Flow of Pagerank with robots.txt and NoFollow, NoIndex

A noindex page can have and accumulate PR.

Dave

[Webnauts]

Originally Posted by kgun

Is that possible, research without a theory?

Kjell, I do base my research on a theory. But once my research starting back in 2006 is completed and been effective, I do not understand what you mean.

[kgun]

I mean that you need a theory to do research.

You can observe reality. You can observe that a stone falls when you drop it, but you need a theory to explain the fall.

Not a big point so continue this important thread.

Originally Posted by Webnauts

Are the bots also white listed? I don't understand.

No. It was wige that indicated that.

[Webnauts]

Originally Posted by crankydave

A noindex page can have and accumulate PR.

Dave

But the PR does not stay on the noindexed page. It passes to other pages through the links of that page. Just to avoid misunderstandings. If you do not agree I have a very simple question.

I had a page that previously did not have the noindex and it had PR 6. Since I noindexed that page and the toolbar is greyed out. And long enough to tell.

Does that make sense?

[Webnauts]

Originally Posted by kgun

I mean that you need a theory to do research.

You can observe reality. You can observe that a stone falls when you drop it, but you need a theory to explain the fall.

Not a big point so continue this important thread.

Excellent! I think the theory is being reveal through our discussion here.

Originally Posted by kgun

No. It was wige that indicated that.

If the bots are white listed and the pages in the protected area have links which may be followed, you do not have dangling/nodes issues.

[crankydave]

Originally Posted by Webnauts

But the PR does not stay on the noindexed page. It passes to other pages through the links of that page. Just to avoid misunderstandings. If you do not agree I have a very simple question.

I had a page that previously did not have the noindex and it had PR 6. Since I noindexed that page and the toolbar is greyed out. And long enough to tell.

Does that make sense?

Naturally the toolbar wouldn't show PR. That doesn't mean the PR coming from links "jumps" to pages it links to. The noindex page accumulates PR it's simply not "seen" via the toolbar. A noindex page accumulates and passes PR in the same way an indexed page would.

ETA... The only way to get the PR to "jump" is to redirect the page.

Dave

[SemAdvance]

Why would anyone believe what Matt Cutts has to say when he is simply a pawn for the largest Propaganda machine that exists today???

Matt cannot tell the truth on anything related to Google, other than who won a Doodle contest.

Lastly the nofollow tag is meant to tell Google you do not trust a site that you link to.....

.why would a webmaster use a nofollow tag when pointing to your own site pages?? That would be saying you do not trust your own site.

That in and of itself should tell everyone the nofollow tag is simple fodder pushed out by the Propaganda Search Engine....ooops I mean Google...

[wige]

To clarify my points about noindex in my previous post...

I believe that "noindex" is something of a misnomer. Specifying that a page is noindex does not actually delete everything about the page.

Lets say you have a source page with five links on it. Further, lets say one of those pages is listed as noindex and disallow in robots.txt. When Google crawls that page, and calculates how much pagerank to assign to each page, it does not consider that one of the pages is listed as noindex/disallow. It simply gives each destination url 1/5 of the source page's pagerank.

Now, when Google finally resolves the URL for the page that is listed as disallow/noindex, Google does not delete the pagerank for the page. After all, Google has to remember that the URL is blocked so that it will not keep trying to crawl that location. It also doesn't delete pagerank - that pagerank gets used in another calculation. The page is still in the index, it is simply hidden so that it never shows up in a user query - including toolbar.google.com queries and site: and other special searches.

[morestar]

Thank you for calling me a highly competent SEO professional Webnauts, you shouldn't have : )

I have to admit, I probably am the most successful SEO in Canada !!!

Not America, but yes in Canada...

And i LOVE Web Pro World!

[morestar]

Originally Posted by SemAdvance

Why would anyone believe what Matt Cutts has to say when he is simply a pawn for the largest Propaganda machine that exists today???

Matt cannot tell the truth on anything related to Google, other than who won a Doodle contest.

Lastly the nofollow tag is meant to tell Google you do not trust a site that you link to.....

.why would a webmaster use a nofollow tag when pointing to your own site pages?? That would be saying you do not trust your own site.

That in and of itself should tell everyone the nofollow tag is simple fodder pushed out by the Propaganda Search Engine....ooops I mean Google...

well...I myself will never use nofollow, however, i can't agree that if you use it on links on your own site it signifies a lack of trust...and if someone uses it on their site on links to other sites, it might mean the user doesn't want to pass PR.

I'm more than sure Google considers "mights" in it's algo more than we think. Take a look at pages with high rankings that have invisible text or text that is hidden or the same color as the background. There must be a "might" in google's thinking like "the developer 'might' have forgotten to apply a color style to the text/p etc.

[kevsta]

interesting discussion but I do agree with Matt C that this kind of "PR sculpting" is very much a second order effect. whilst it may make small differences it's kind of like picking over the scraps on the floor while missing the steak on the table.

[Webnauts]

Originally Posted by wige

To clarify my points about noindex in my previous post...

I believe that "noindex" is something of a misnomer. Specifying that a page is noindex does not actually delete everything about the page.

Lets say you have a source page with five links on it. Further, lets say one of those pages is listed as noindex and disallow in robots.txt. When Google crawls that page, and calculates how much pagerank to assign to each page, it does not consider that one of the pages is listed as noindex/disallow. It simply gives each destination url 1/5 of the source page's pagerank.

1. You cannot have noindex and nofollow the same time in the robots.txt.

Andy Beard backs that up with his comment here SEOmoz | Headsmacking Tip #13: Don't Accidentally Block Link Juice with Robots.txt

"Google ranks pages blocked with Robots.txt much lower than they used to just 12 months ago, even if they have tons of authority links.
They will appear in very long tail searches, and site searches.

"However just because they don't appear in search results doesn't mean they aren't sucking juice out of your site.
If you use noindex, don't forget to remove the disallow in robots.txt, otherwise Google won't see the noindex."

It gives 1/5 to the link but not to the noindexed destination page its self. When the bot will arrive to the noindexed page it will go on carrying the PR and pass it to the the pages I link from there. Otherwise, why should dangling/node pages be a problem? If that would not be a problem, then I can also add the nofollow meta tag on the noindex page too, or?

Originally Posted by wige

Now, when Google finally resolves the URL for the page that is listed as disallow/noindex, Google does not delete the pagerank for the page. After all, Google has to remember that the URL is blocked so that it will not keep trying to crawl that location. It also doesn't delete pagerank - that pagerank gets used in another calculation. The page is still in the index, it is simply hidden so that it never shows up in a user query - including toolbar.google.com queries and site: and other special searches.

Google does not delete the PR? OK. Then tell me this. How much PR does a page need to have so it will be white and not grey.

I think there is a misconcept here. Disallow and Noindex are two entirely different things. Disallow blocks and noindex does not block. Noindex allows Google to crawl through the page, but it advises Google not to index it.

The noindex page does not stay in the index. It is removed entirely! That is fact.

Back to the quotation Dave posted above:

Eric Enge Can a NoIndex page accumulate PageRank?

Matt Cutts A NoIndex page can accumulate PageRank, because the links are still followed outwards from a NoIndex page.

Eric Enge So, it can accumulate and pass PageRank.

Matt Cutts Right, and it will still accumulate PageRank, but it won’t be showing in our Index. So, I wouldn’t make a NoIndex page that itself is a dead end. You can make a NoIndex page that has links to lots of other pages.
Eric Enge Interviews Google’s Matt Cutts

- Can someone explain me what the word accumulate mean?
- Can someone explain me why I shouldn't have a dead end?

[Webnauts]

Originally Posted by kevsta

interesting discussion but I do agree with Matt C that this kind of "PR sculpting" is very much a second order effect. whilst it may make small differences it's kind of like picking over the scraps on the floor while missing the steak on the table.

I do not agree Ken. Because what we are discussing here is not really PageRank Sculpting. I would call that bots herding. It is about showing Google what pages you feel are important and which not. And if not important pages have IBLs carrying PR, that they will pass that PR to the important pages. And when or where Matt Cutts talked about what I am talking about here? That is new to me.

[kevsta]

Originally Posted by Webnauts

I do not agree Ken. Because what we are discussing here is not really PageRank Sculpting. I would call that bots herding. It is about showing Google what pages you feel are important and which not. And if not important pages have IBLs carrying PR, that they will pass that PR to the important pages. And when or where Matt Cutts talked about what I am talking about here? That is new to me.

"bot herding" for what purpose exactly?

..MC very famously said he wouldnt worry about trying to sculpt PR because it is a second order effect, and better links will be a better solution every time. will find the link to the interview if youve forgotten?

[wige]

Originally Posted by Webnauts

Google does not delete the PR? OK. Then tell me this. How much PR does a page need to have so it will be white and not grey.

Well, remember, toolbar pagerank is not the same thing as internal pagerank. Just because a page doesn't have a toolbar pagerank does not mean that the page has no internal pagerank.

Toolbar pagerank information is stored in a seperate server from Google's actual indices. The tbpr database is built periodically, and Google has mentioned in the past certain sites are omittted when the tbpr database is built. I firmly believe that pages marked as noindex are simply left out of the tbpr database, rather than the omission indicating the URLs have been removed from the index of known urls.

There is no way to confirm this absolutely without working at Google, but to me, it seems that the database would be less efficient if Google removed noindex pages from the index instead of hiding them - it would throw off the calculations of pagerank (since in theory pagerank can't be destroyed), it would affect the operation of spiders (since they would in effect keep forgetting that the page is not indexable).

The noindex page does not stay in the index. It is removed entirely! That is fact.

This is theory. How can it be proven? And as noted above, not being in the tbpr database does not prove it.

[kevsta]

Originally Posted by crankydave

A noindex page accumulates and passes PR in the same way an indexed page would.

Dave

this was my understanding of this too

[Webnauts]

Originally Posted by kevsta

"bot herding" for what purpose exactly?

..MC very famously said he wouldnt worry about trying to sculpt PR because it is a second order effect, and better links will be a better solution every time. will find the link to the interview if youve forgotten?

1. Fall weather forecast
2. SMX Video - Matt Cutts Explains How to Get Out of Google’s Supplemental Index

And yes. Please find that link if you can.

Thanks.

[Webnauts]

Originally Posted by kevsta

"bot herding" for what purpose exactly?

For example Bot Herding = Bot Herding

[Webnauts]

Originally Posted by wige

Well, remember, toolbar pagerank is not the same thing as internal pagerank.

Just because a page doesn't have a toolbar pagerank does not mean that the page has no internal pagerank.

Toolbar pagerank information is stored in a seperate server from Google's actual indices. The tbpr database is built periodically, and Google has mentioned in the past certain sites are omittted when the tbpr database is built.

That is clear. But if I noindex a page that had PR 6 and becomes grey and the links pointing to that pages exist for lets say a year, should that PR show up some time?

Originally Posted by wige

I firmly believe that pages marked as noindex are simply left out of the tbpr database, rather than the omission indicating the URLs have been removed from the index of known urls.

If you are right, then the toolbar values are not just only out-of-date (2-4 months old), but they are faked!!!

This deserves a thread of its own!

Originally Posted by wige

There is no way to confirm this absolutely without working at Google, but to me, it seems that the database would be less efficient if Google removed noindex pages from the index instead of hiding them

Google doesn't delete information in general. Deletion for Google means caching or saving into not accessible to the public areas.

Originally Posted by wige

- it would throw off the calculations of pagerank (since in theory pagerank can't be destroyed), it would affect the operation of spiders (since they would in effect keep forgetting that the page is not indexable).

To remember those pages do those pages need to have PageRank?

Originally Posted by wige

This is theory. How can it be proven? And as noted above, not being in the tbpr database does not prove it.

Well as I said if the noindex page has PageRank and it is not displayed in the toolbar, the toolbar values are fake. If that is they case, the toolbar sucks more than my imagination could perceive since I am working as an SEO professional.

[kevsta]

Originally Posted by Webnauts

1. Fall weather forecast
2. SMX Video - Matt Cutts Explains How to Get Out of Google’s Supplemental Index

And yes. Please find that link if you can.

Thanks.

Does PR Sculpting Work?

http://www.jaankanellis.com/update-o...nd-matt-cutts/

He used his analogy again to help explain it:

Nofollowing your internals can affect your ranking in Google, but it’s a 2nd order effect.
My analogy is: suppose you’ve got $100. Would you rather work on getting $300, or would you spend your time planning how to spend your $100 more wisely.

Spending the $100 more wisely is a matter of good site architecture (and nofollowing/sculpting PageRank if you want). But most people would benefit more from looking at how to get to the $300 level.

and went on to say that PR Sculpting is just another small piece that can help some websites.

[Terry Van Horne]

As to the way PR is treated on noindex pages etc. I would tend to lean toward agreeing with Cranky Dave. As to htaccess on Windows, We use ISA server to control entry to our sites and network and before that came along I basically secured everything with authentication as there weren't many easy options. I strongly recommend this because using Robots.txt and the the other protocal methods are wonky. If you want it to be "for sure" use authentication. Even that is wonky now Google has decided to go through forms, in fact when they announced they would be doing that it was first thing that came to my mind.

To some degree I agree with semadvance about the nofollowing your site, however, I belive there are some cases where it may be wise to do so. For instance when you link from a clients site to your site. In both the html5 RFC for rel=nofollow and to a lesser degree Google guidelines that is a link where the primary consideration for the link is commercial so it should be nofollowed. As a webmaster I've chosen to link to site both with and without nofollow on the same page. On SeoPros we decided to nofollow sponsorships and leave the directory listings where the review is first and foremost with no rel=. It still isn't cut and dry but it is IMO, a little clearer with the addition to the HTML5 RFC.

[Webnauts]

Originally Posted by kevsta

Does PR Sculpting Work?

http://www.jaankanellis.com/update-o...nd-matt-cutts/

If it is posed that way, I have say that I never saw a web site on the Internet that has the same good architecture and navigation. I am not talking about visual appealing, as it is not the beauty.

Does that mean that it is not worth to try to herd the bots pass the PR to my important pages and advise them to ignore the non important pages like login, signup, subscribe, etc?

So should we stick to the old school methodologies, disallowing those pages with a disallow directive n the robots.txt? Or should we share our PR with those pages.

We are all posting articles and videos of MC here, but did someone try to diff them? I mean compare with each other (for the less technical members).

Another question:

I have internal anchors on each page, i.e skip links for blind users, etc. like this:

<a href="#main-content" accesskey="2">Skip to main content</a>

Does Google assign Pagerank to those internal anchors?

Next question:

I am on the services age of my site. For an appropriate navigation I need to have 3 links to my homepage.

1. In the logo.
2. In the main navigation.
3. In the breadcrumbs.

Now. If 3 links are pointing to the same page, won't I have a leak of PR?

That is all I would like to know, and then I am coming with more details.

Thanks.

[Webnauts]

Originally Posted by Terry Van Horne

To some degree I agree with semadvance about the nofollowing your site, however, I belive there are some cases where it may be wise to do so. For instance when you link from a clients site to your site.

Until recently I was a nofollow enemy. Everyone knows that here. But that said, does not mean that I am a fan. I see after the last years of research that it would be wise to use in some cases.

I will go into details later in the thread. It will be about this New Canonical Tag from the big 3 but at an extended level.

Thanks Terry for dropping by.

[Terry Van Horne]

Webnauts, IMO, the pages you are trying to tell the bots to ignore, you're preaching to the choir, they know they aren't important by the location on your page/navigation/ link hierarchy and the number of IBLs to them. It's not co-incidence that directories with site links almost always include the submission page. Figuring out the most Important pages is what the PR algo is all about. Trying to make the good pages look better is IMO, a waste of time and resources, could easily bite you in the a$$e too if engines began to take the implied trust literally as semadvance pointed out earlier.

[kevsta]

Originally Posted by Webnauts

Next question:

I am on the services age of my site. For an appropriate navigation I need to have 3 links to my homepage.

1. In the logo.
2. In the main navigation.
3. In the breadcrumbs.

Now. If 3 links are pointing to the same page, won't I have a leak of PR?

That is all I would like to know, and then I am coming with more details.

Thanks.

yes, well, not a "leak" of PR but as G only counts the first link it's obviously not ideal.

however nofollowing the wrong 2 will hurt you more than help because even if the link's nofollowed, G may count that link's anchor text, not the one you want them to, then ignore the next 2 links completely

[kevsta]

Originally Posted by Terry Van Horne

Webnauts, IMO, the pages you are trying to tell the bots to ignore, you're preaching to the choir, they know they aren't important by the location on your page/navigation/ link hierarchy and the number of IBLs to them.

I agree with this too, I think G are pretty good at working out what's important and what isnt. most contact pages I see are grey TB even if they're sitewide linked to, as are log in pages etc. they may have some small amount of real PR but all you need to do is manage the links away from them correctly and you can let it flow and not worry IMO.

[kgun]

First of all, I am not in the SEO business, so this may seem like stupid remarks. And my questions / remarks are about external links. I am thinking loud to get your reactions.

Originally Posted by kgun

Is this wrong Books on Web 2.0 developement, explaining CSS, XML, DOM, AJAX, OOP and Design patterns.

.htaccess

order deny,allow
deny from all

except white listed IP's in the /Books/ folder ?

1. Will the white list create problems?
2. More precisely: Can Bot's get an URI reference or a dangling node (link) via the white list?

If a person on the white list links to a page in the protected area (why should (s)he want to do that? - On an assumption that the protected area will be allowed later?) from hers / his site with a semantic anchor text, the link may be indexed, but it is still broken (dead). How probable is such a broken link that degrades that external page?

Originally Posted by Webnauts

Using .htaccess to deny to a folder and to sub folder is a bad thing to do. I mean if you are returning a 403. That is because your are creating a dangling/node page, which are otherwise called dead end pages/folders.

Is that different from what Matt Cutts put a on in the video linked to in your first post? Why does he not say that you shall redirect the login in page? Is that self evident?

Originally Posted by Webnauts

If the bots are white listed and the pages in the protected area have links which may be followed, you do not have dangling/nodes issues.

So you recommend a redirect or is the new solution acceptable?

[monfis]

[quaote]If you are right, then the toolbar values are not just only out-of-date (2-4 months old), but they are faked!!![/quote]

My observations quering Google datacenters to check PR and observing certain toolbar values suggest, they are often fake. I am just not sure about the rules to show or not to show a fake PR.

[Webnauts]

Originally Posted by kevsta

yes, well, not a "leak" of PR but as G only counts the first link it's obviously not ideal.

however nofollowing the wrong 2 will hurt you more than help because even if the link's nofollowed, G may count that link's anchor text, not the one you want them to, then ignore the next 2 links completely

Thank you for the link. I already know this story, but I am not wondering who is right: Google passes second link’s anchor text | SEO Theory - SEO Theory and Analysis Blog

I am still wondering if I have 3 links to the same destination, won't their power be weaken?

So I am wondering if the second and third link is ignored by Google, then they do not need to be nofollowed, or?