View Full Version : Bot Herding: The Ultimate Tool for PageRank Sculpting
Webnauts
05-12-2009, 05:59 PM
I would appreciate your thoughts: Bot Herding: The Ultimate Tool for PageRank Sculpting (http://www.searchenginejournal.com/bot-herding-pagerank-sculpting/10352/)
I am sure by implementing the above techniques, you can achieve the maximum possible control over the PageRank flow within your web site. How about you? ;)
The Leader
05-12-2009, 07:15 PM
Sounds to complicated to me!
It'll be quicker for me to build links instead.
bobitza
05-12-2009, 08:36 PM
Good stuff John, thanks for sharing.
I sort of disagree on the use of nofollow ... perhaps because I am already sculpting PR with nofollows, hehe. No, seriously, if I no follow my TOS page I don't think someone will be stupid enough to bother linking to that.
The noindex directive for robots.txt is new to me so again, thanks for sharing.
Webnauts
05-13-2009, 04:46 AM
I think you both must re-read the article. I say so, because if you really understood it, you would have posted something entirely different. :)
earnest
05-13-2009, 05:06 AM
Thanks for the idea.
Feel free to call me a total noob, but the fact that pagerank was assigned to non-indexed pages was breaking news for me. Hence the question: what about internal links?
bobitza
05-13-2009, 09:32 AM
I think you both must re-read the article. I say so, because if you really understood it, you would have posted something entirely different. :)
OK, let me take another shoot at it :)
So, you say ...
Page A is linking to page B, and that link is attributed nofollow status. In this case Page A will not pass PageRank to Page B, right? But what is if page B is linked from another page of your web site or from an external web site without that link being protected with the nofollow attribute? Won’t PageRank be assigned to page B? Won’t a snippet show up in the search results? It will! Is that what you want? Wouldn’t you prefer to make sure that the incoming PageRank for page B is being passed to the most important web pages of your web site?
Of course I want the incoming PR to be assigned to most important web pages; and I will never (to stick to my previous example with the TOS page) create linking campaigns to my TOS page.
In the scenario that you described, if someone links to my B page (aka TOS page), this is something that I can not control. Because on my part I nofollowed all the internal links, the PR that is assigned to page B is coming ONLY from other websites, right? Yeah, I know, crazy people do crazy things (like linking to your TOS page without asking).
If I have outgoing internal or external links on Page B that will benefit from the little PR of the page, that's even better!
You also say ...
if Page B is linked from an external site without being protected by the nofollow attribute, PageRank will still be assigned to it, and it can still show up in the search results.
... and that's bad because ... ? I mean, if I focus my seo and linking efforts to page A, I don't think page B will be in the top 100 results of anything important.
abbyink
05-13-2009, 11:29 AM
Does anyone know if the NoIndex directive for robots files is slated to become a standard for bots?
Webnauts
05-13-2009, 01:12 PM
Does anyone know if the NoIndex directive for robots files is slated to become a standard for bots?
Google supports the robots.txt noindex directive. All other engines not.
SteveGerencser
05-14-2009, 11:10 PM
Ok, I read that post a few times and think that the basic take away from it is this -
You have a page (A) with a noindex on it to keep Google from indexing the page..
Page (A) links out to page (B)
Now we have a third page (C) and it links to page (A).. But, since page (A) is noindexed you are saying that the pagerank that would have been passed from (C) to (A) is now passed from (C) to (B), skipping over (A)..
Is this pretty much what you are saying in that blog post??
bobitza
05-15-2009, 09:28 AM
Now I'm really confused ...
John, you might wanna re-phrase that post.
Webnauts
05-20-2009, 10:33 PM
... and that's bad because ... ? I mean, if I focus my seo and linking efforts to page A, I don't think page B will be in the top 100 results of anything important.
My question is, why should incoming PR to page B be wasted and not distributed to other pages where you need PR?
Webnauts
05-20-2009, 10:34 PM
Ok, I read that post a few times and think that the basic take away from it is this -
You have a page (A) with a noindex on it to keep Google from indexing the page..
Page (A) links out to page (B)
Now we have a third page (C) and it links to page (A).. But, since page (A) is noindexed you are saying that the pagerank that would have been passed from (C) to (A) is now passed from (C) to (B), skipping over (A)..
Is this pretty much what you are saying in that blog post??
Not at all. I would suggest you to read it again.
Webnauts
08-13-2009, 03:23 PM
Sounds to complicated to me!
It'll be quicker for me to build links instead.
Quicker to build links instead? How can you do that so quickly. With spamming?
VideoWhisper.com
08-15-2009, 12:59 PM
This is for prioritizing pages on a site. It will also need external back links to these internal pages to get these ranked well in results. Because prioritization is useless if there's not search engine traffic to prioritize.
A more drastic method would be to block crawlers from indexing pages you don't consider important. Like member profiles in a forum.
GameFriends
08-31-2009, 11:34 PM
Create a sitemap/footer or have a universal header link on your website (most do by now) linking to your most important pages. This will eliminate the "Dangling link" since at least one link/page will be ALWAYS be passed PR from the sitemap/footer/header. Simply no follow the links on the sitemap/footer/header you don't care about and the rest takes care of itself.
Problem solved.
Webnauts
09-01-2009, 04:39 AM
Create a sitemap/footer or have a universal header link on your website (most do by now) linking to your most important pages. This will eliminate the "Dangling link" since at least one link/page will be ALWAYS be passed PR from the sitemap/footer/header. Simply no follow the links on the sitemap/footer/header you don't care about and the rest takes care of itself.
Problem solved.
That is fully outdated, incorrect and misleading information!
Middle of June this year Google officially announced that the nofollow attribute for internal linking does not work. And that already for almost a year, but they did not inform the public earlier.
That is fully outdated, incorrect and misleading information!
Middle of June this year Google officially announced that the nofollow attribute for internal linking does not work. And that already for almost a year, but they did not inform the public earlier.
My bolding.
But what about
<meta name="robots" content="index,follow">
on your sitemap (page / file) and
<meta name="robots" content="index,nofollow">
on all other pages?
That is how I have done it on my main site, second link in my signature.
From your original post:
Bot Herding: The Ultimate Tool for PageRank Sculpting (http://www.searchenginejournal.com/bot-herding-pagerank-sculpting/10352/)
Matt Inertia says
I’ve done basic sculpting on small sites and it’s made no difference but i can see this being more of an issue on huge sites.
I see that you modified your original post to be about pagerank flow and not sculpting, but Matt may have a point about small sites or ... ?
And the last line in my robots.txt file is:
Sitemap: http://www.multifinanceit.com/sitemap.htm
since I made my own sitemap in HTML, but HTML is XML, so that should not be a problem.
Webnauts
09-01-2009, 11:55 AM
My bolding.
But what about
<meta name="robots" content="index,follow">
on your sitemap (page / file)
You do not need that.
<meta name="robots" content="index,nofollow">
on all other pages?
Then you will create nodes (dangling pages)
That is how I have done it on my main site, second link in my signature.
From your original post:
Bot Herding: The Ultimate Tool for PageRank Sculpting (http://www.searchenginejournal.com/bot-herding-pagerank-sculpting/10352/)
To be honest the reason was, that it was never about PageRank Sculpting. I was about bot herding, but which was unknown, so lets say it was a sort of link baiting. :)
Matt Inertia says
I’ve done basic sculpting on small sites and it’s made no difference but i can see this being more of an issue on huge sites.
I see that you modified your original post to be about pagerank flow and not sculpting, but Matt may have a point about small sites or ... ?
I do not know how Matt came up with that. Maybe because many guys are preaching that? Well I feel sorry for those, but I was not expecting to hear something different, when Google suddenly said: Nofollow for PageRank Sculpting, No Go Gentlemen. :lol:
To resume, my answer is very simple: Every drop of juice is important to me.
And the last line in my robots.txt file is:
Sitemap: Site Map (http://www.multifinanceit.com/sitemap.htm)
Thats fine.
You do not need that.
I know the defaults Jon. Only for illustrative purposes.
Then you will create nodes (dangling pages)
Or leaf nodes in DOM web page terminology:rolleyes: But the web is a graph and not a tree, so it may be compared to a broken bridge ... ??
To resume, my answer is very simple: Every drop of juice is important to me.
I gave too much away, now I may be giving too little away. Time will show what technology I end up with.
Webnauts
09-03-2009, 08:23 AM
I know the defaults Jon. Only for illustrative purposes.
If a beginner would read that, would definetely not perceive that as posted for illustrative purposes.
Or leaf nodes in DOM web page terminology:rolleyes: But the web is a graph and not a tree, so it may be compared to a broken bridge ... ??
When did I claim the oposite?
I gave too much away, now I may be giving too little away. Time will show what technology I end up with.
You gave too much? I think you should should give away misleading iformation Kjell. If you call that much, don't give away anything. ;)
If a beginner would read that, would definetely not perceive that as posted for illustrative purposes.
And the beginner would be just fine, aside from an extra line of markup. Period.
You gave too much? I think you should should give away misleading iformation Kjell. If you call that much, don't give away anything. ;)
Too much linkjuice. Isn't that my decision?
Misleading information?
Where have I given out misleading information? If you can not write anything on this forum for illustrative purposes, that may be worse to a beginner.
And you will find those tags on my sites. Those issues are not my main activity at the moment.
The Leader
09-05-2009, 04:39 AM
So if we don't want to pass pr juice from one page to another are we saying that the first page loses page rank and the other page gains page rank?
watto
09-07-2009, 11:46 PM
What exactly do you call a big site?
Webnauts recently developed a robots.txt file for my site www .businessesforsales.com to herd the bots to the important pages.
Without any link building what so ever, my rankings (as of today) have jumped from position #88 in google to position #6 for one of my main terms 'businesses for sale'. I can't believe it!
I am still in shock as I never expected such results in such a short time. The robots.txt file has been in place for approx 3-4 weeks.
What are your thoughts?
watto
WhitneySegura
09-08-2009, 12:53 AM
Well, all this discussion about bots herding is turning me on!
Watto is absolutely correct and telling the truth about his results, as not only did I watch as John worked on his website and his rankings shoot up, but I had John do some work for my stores and throw up a robots file.
Originally, I was very skeptical to jump into something that hasn't been brought mainstream and I asked Watto, "Hey man, I am not sure how this is going to affect my current rankings and I was wondering what your experience has been?" He told me that he felt the exact same way before he did it, but immediately he convinced me that this was like crack for the SERPs.
Anyways, we implemented bots herding about 2 weeks ago and every single one of my 1st and 2nd tier keywords had dramatically increased. At the time I was not even thinking about ranking for my now primary keyword target, which is also the top search term in my market.
I went from around #75-80 down to #40 within 1 week, throughout the past week I shot up to #24 then about 2 days ago I hit #19, and today I am very pleased to say I am already #14.
My website is not even 7 months old ranking for such kind of terms, its truly incredible. John, you my friend are truly a professional and I would recommend your services to anyone & everyone.
Honestly, I can say from my own experience and the experience of watching my friends website rankings, that bot herding IS the ultimate form of page rank sculpting.
End of story, tell me whats better...? Please.