Tagging irrelevant page areas, rel="nofollow" etc.

[kgun]

1. Background:

1. http://www.webproworld.com/google-di...tml#post437745
2. http://www.webproworld.com/search-en...tml#post403724
3. http://www.webproworld.com/search-en...tml#post442147

I found some interesting articles on this

Smart IT Consulting Services

(
Related blog: sebastianx.blogspot.com

Example: Just another victim of the nofollow plague
)

site related to those subjects. Here are some relevant snippets.

2. Tagging irrelevant page areas: class=robots-nocontent

"Telling a search engine that particular page areas aren't related to a page's core contents was a problem, until Yahoo! introduced the "robots-nocontent" class name in May 2007. Perhaps other search engines will follow and support this mechanism too. Google has something similar called section targeting for the AdSense crawler, but puts crawler directives in HTML comments instead of the class attribute".

Source: Tagging irrelevant page areas: class=robots-nocontent

3. Link Specific Regulation: REL=NOFOLLOW

Google has introduced the NOFOLLOW value in the <A> tag's REL attribute in January, 2005, as an instrument to prevent comment spam. Yahoo, MSN and guestbook/forum/blog software makers quickly joined the initiative. In the meantime this syntax found its way into Google's guidelines and Google reps encourage webmasters to use it anytime they can't vouch for a link.

......................

There are good reasons not to use REL=NOFOLLOW to hoard PageRank™. First, PageRank™ hoarding is easy to discover and you will earn a ranking penalty for over-optimizing. Second, other webmasters are smart too and will cancel link trades if you cheat.

Source: Link Specific Regulation: REL=NOFOLLOW

4. URL Specific Control: the Robots META Tag

Note that the robots META tag is for use in HTML documents only. If you offer your content additionally in PDF or DOC format, and you don't want to find the PDF/DOC-files in search results, store them in a directory protected by robots.txt or disallow these extensions in general.

Source: URL Specific Control: the Robots META Tag

5. Steering and Supporting Search Engine Crawling

Source: Steering and Supporting Search Engine Crawling

6. Identifying and Tracking SE Crawling

An interesting PHP function:

function isSpider ( $userAgent ) {
    if ( stristr($userAgent, "Googlebot")    || /* Google */
         stristr($userAgent, "Slurp")    || /* Inktomi/Y! */
         stristr($userAgent, "MSNBOT")    || /* MSN */
         stristr($userAgent, "teoma")    || /* Teoma */
         stristr($userAgent, "ia_archiver")    || /* Alexa */
         stristr($userAgent, "Scooter")    || /* Altavista */
         stristr($userAgent, "Mercator")    || /* Altavista */
         stristr($userAgent, "FAST")    || /* AllTheWeb */
         stristr($userAgent, "MantraAgent")    || /* LookSmart */
         stristr($userAgent, "Lycos")    || /* Lycos */
         stristr($userAgent, "ZyBorg")    /* WISEnut */
    ) return TRUE;
    return FALSE;
}

if (isSpider(getenv("HTTP_USER_AGENT"))) {
    $useSessionID = FALSE;
    $logAccess = TRUE;
} 

Source: Identifying and Tracking SE Crawling

7. Why is there not a rel="follow" attribute?

The meta nofollow tag in the head element refers to all links in the page.

http://www.google.com/support/webmasters/bin/answer.py?answer=33581&t...

If you put after that rel="follow" for individual links either it will not work or it will confuse the bot.

Source:
Google Groups Google Webmaster Help discussion thread: rel="follow" ? - Crawling, indexing, and ranking | Google Groups

8. A guide to clever linking for geeks and savvy programmers

Anatomy and Deployment of Links

9. Links for the future?

Advanced semantic linking and transclusion.

[Doc]

Kjell, until I joined WPW, I was blissfully unaware of the level of my ignorance. This is a particular area in which I feel more than a little confused.

Certainly part of that confusion comes from the differing opinions on the validity of nofollows, and even the possible penalties for using them. Traditionally, I have "trusted my gut" on issues I couldn't define clearly. And this is one where I feel safer in simply ensuring that everything I build is worthy of scrutiny, or I'll do without it.

At least until I've accrued a lot more expertise than I have now. At times, it seems as though the mines in this field are placed too closely to allow a safe footprint.

[Webnauts]

1. Background:
I found some interesting articles on this

Smart IT Consulting Services

site related to those subjects. Here are some relevant snippets.

I do not understand why are you linking to a guy that does not even have basic knowledge of SEO.

He says we should add on our pages:

<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">

Is that a joke? [/quote]

2. Tagging irrelevant page areas: class=robots-nocontent

"Telling a search engine that particular page areas aren't related to a page's core contents was a problem, until Yahoo! introduced the "robots-nocontent" class name in May 2007. Perhaps other search engines will follow and support this mechanism too. Google has something similar called section targeting for the AdSense crawler, but puts crawler directives in HTML comments instead of the class attribute".

Source: Tagging irrelevant page areas: class=robots-nocontent

Very old news and useless. Only works with Yahoo.

3. Link Specific Regulation: REL=NOFOLLOW

Google has introduced the NOFOLLOW value in the <A> tag's REL attribute in January, 2005, as an instrument to prevent comment spam. Yahoo, MSN and guestbook/forum/blog software makers quickly joined the initiative. In the meantime this syntax found its way into Google's guidelines and Google reps encourage webmasters to use it anytime they can't vouch for a link.

......................

There are good reasons not to use REL=NOFOLLOW to hoard PageRank™. First, PageRank™ hoarding is easy to discover and you will earn a ranking penalty for over-optimizing. Second, other webmasters are smart too and will cancel link trades if you cheat.

Source: Link Specific Regulation: REL=NOFOLLOW

Ancient news too. What point do you want to make here?

4. URL Specific Control: the Robots META Tag

Note that the robots META tag is for use in HTML documents only. If you offer your content additionally in PDF or DOC format, and you don't want to find the PDF/DOC-files in search results, store them in a directory protected by robots.txt or disallow these extensions in general.

Source: URL Specific Control: the Robots META Tag

A better solution for blocking PDFs is using X-Robots noindex, so the crawlers will crawl and not index.

5. Steering and Supporting Search Engine Crawling

Source: Steering and Supporting Search Engine Crawling

I had no time to check this link.

6. Identifying and Tracking SE Crawling

An interesting PHP function:

function isSpider ( $userAgent ) {
    if ( stristr($userAgent, "Googlebot")    || /* Google */
         stristr($userAgent, "Slurp")    || /* Inktomi/Y! */
         stristr($userAgent, "MSNBOT")    || /* MSN */
         stristr($userAgent, "teoma")    || /* Teoma */
         stristr($userAgent, "ia_archiver")    || /* Alexa */
         stristr($userAgent, "Scooter")    || /* Altavista */
         stristr($userAgent, "Mercator")    || /* Altavista */
         stristr($userAgent, "FAST")    || /* AllTheWeb */
         stristr($userAgent, "MantraAgent")    || /* LookSmart */
         stristr($userAgent, "Lycos")    || /* Lycos */
         stristr($userAgent, "ZyBorg")    /* WISEnut */
    ) return TRUE;
    return FALSE;
}

if (isSpider(getenv("HTTP_USER_AGENT"))) {
    $useSessionID = FALSE;
    $logAccess = TRUE;
} 

Source: Identifying and Tracking SE Crawling

Is that the joke of the year? I can easily fake the user agents.

7. Why is there not a rel="follow" attribute?

The meta nofollow tag in the head element refers to all links in the page.

http://www.google.com/support/webmasters/bin/answer.py?answer=33581&t...

If you put after that rel="follow" for individual links either it will not work or it will confuse the bot.

Source:
Google Groups Google Webmaster Help discussion thread: rel="follow" ? - Crawling, indexing, and ranking | Google Groups

A rel="follow" is not necessary, as links are followed by default.

8. A guide to clever linking for geeks and savvy programmers

Anatomy and Deployment of Links

I had no time to check this link.

9. Links for the future?

Advanced semantic linking and transclusion.

I had no time to check this link. But reading the title tells me that it is nothing new to me.

So what point are you trying to make? Can you make that short and good without bombing the thread with links, and most with inaccurate or obsolete information?

Thanks man.

[kgun]

And this is one where I feel safer in simply ensuring that everything I build is worthy of scrutiny, or I'll do without it.

At least until I've accrued a lot more expertise than I have now. At times, it seems as though the mines in this field are placed too closely to allow a safe footprint.

Perhaps the best strategy.

[ogletreeseo]

Don't forget a sitemap is unnecessary if you have your internal linking set up properly. The Google sitemap thing is equal to search engine submission. It is pointless. If the sitemap is the only way google is going to find your pages then that page is not going to rank for anything anyhow.

[kgun]

@John / Webnauts

an advice that you can listen to or not.

Don't draw too fast conclusions and be careful when you judge other people that may know as much as you.

And what is self evident to you need not be it to other members or the general surfer.

You and I have had some disagreements here at WPW. Here http://www.webproworld.com/search-en...tml#post462724 is the most recent.

And you should be more careful. Much could have been said and concluded based on this Bot Herding: The Ultimate Tool for PageRank Sculpting | Search Engine Journal article.

I will not.

The start of your answer indicates that you have not read much of the external articles linked to in my op.

Example much have been written here about PageRank sculpting that it seems that you have finally understood. Not so much have been written about PageRank hoarding. Is the statement in red in my first post correct or not?

I look forward to constructuve feedback

[kgun]

Don't forget a sitemap is unnecessary if you have your internal linking set up properly. The Google sitemap thing is equal to search engine submission. It is pointless. If the sitemap is the only way google is going to find your pages then that page is not going to rank for anything anyhow.

I have not cared so much about sitemaps. I have one made in html on my biggest site, the second link in my signature. It is made for the user and bots that need it.