Block spiders from PART of a page?

[davebarnes]

Is it possible to block/prevent search engines from spidering PART of a page?

I know how to seal off part/all of a website. My question here is: if I let a search engine spider get to a page can I separate the page into 2 parts: allowed for spidering and DISallowed?

If yes, how do I accomplish that?

thanks,
dave

[Feydakin]

You could put that part of the page in a javascript document.write command.. Supposedly SEs can't read javascript..

Now, for the more interesting part, why would you want to do this??

[sunbedkid]

The only reason I can see for him wantiing to do this, is to have the pagerank for a links page, then not give away any link juice to potential link partners.

Very sneeky!!

[sorahayda]

using rel=no follow attribute or robot txt

[Webnauts]

Originally Posted by sorahayda

using rel=no follow attribute or robot txt

I hope this was a joke buddy. I think you should re-read the question.

[Webnauts]

Originally Posted by davebarnes

Is it possible to block/prevent search engines from spidering PART of a page?

I know how to seal off part/all of a website. My question here is: if I let a search engine spider get to a page can I separate the page into 2 parts: allowed for spidering and DISallowed?

If yes, how do I accomplish that?

thanks,
dave

Dave we created a php script a while ago, where we could hide efficiently parts of our pages content. We took it down since Google announced that they perceive such as clocking and that they can spot such stuff with diffing using MD5 checksum.
I still don't believe that they really can, but we want to be on the safe side just in case.

About using Javascript I would also be careful. It is not 100% accurate information, that Google cannot read JavaScript at all.

Would you mind being more specific what exactly your problem is, so I can see if we can come up with a possible legitimate solution?

[Webnauts]

Originally Posted by sunbedkid

The only reason I can see for him wantiing to do this, is to have the pagerank for a links page, then not give away any link juice to potential link partners.

Very sneeky!!

I do not believe Dave would do such stuff buddy. Don't you think you are getting sort of offensive here?
Dave is a senior and highly respected member and professional here at WPW. So please think before you post.

Thanks,

John

[kgun]

Is it possible to use AJAX technology, the XMLHttpRequest object and / or iFrames?

[Webnauts]

Originally Posted by kgun

Is it possible to use AJAX technology, the XMLHttpRequest object or iFrames?

For both options if accessibility is not a concern, then should not be a problem. Just in case we stick to the iframes, notice that they are not supported in XHTML Strict. Otherwise will be ok.

[crankydave]

That was my initial thought too... iFrames.

Depending upon what it is, what about making it an image? If it was a chart or a block of text, put it in an image.

Dave

[davebarnes]

Here is the page in question: Creative Loafing Tampa | Food & Drink | Features | It took a Village

Here is the problem.
I want the search engines to focus on indexing the contents of: the main article and the 3 blocks on the right-hand side as these are all about the restaurant being reviewed and other food and drink.
But, right now, the spiders also see all pull-down lists in the box at the top. These items are detracting from the FOCUS of the article.

In other words, I want the spider to visit and leave with thoughts of "wine and homogenous American fine dining" in its head and not with "Ethiopian, Filipino, Fondue, French" and all the other lists contained in the Ajax construction.

[crankydave]

Why not deliver that nav in an iFrame?

Dave

[Webnauts]

Originally Posted by davebarnes

Here is the page in question: Creative Loafing Tampa | Food & Drink | Features | It took a Village

Here is the problem.
I want the search engines to focus on indexing the contents of: the main article and the 3 blocks on the right-hand side as these are all about the restaurant being reviewed and other food and drink.
But, right now, the spiders also see all pull-down lists in the box at the top. These items are detracting from the FOCUS of the article.

In other words, I want the spider to visit and leave with thoughts of "wine and homogenous American fine dining" in its head and not with "Ethiopian, Filipino, Fondue, French" and all the other lists contained in the Ajax construction.

Dave, no need to worry about that. See here what the bots see here:
Free SEO Software Tool & Text Browser, Search Engine Optimization Tools - SEO Browser

But I am very concerned about the use of display: none. I am sure you are not trying to deceive the SE but it is very possible that they will perceive it that way.

Here is a tool to check this out: Search engine SPAM detector

[crankydave]

John...

Google is caching the text in the dropdowns so I believe they are seeing it...

Creative Loafing Tampa | Food & Drink | Features | It took a Village

Dave

[activeco]

Originally Posted by crankydave

Why not deliver that nav in an iFrame?

Dave

Exactly Dave & Kgun.
Put everything what you don't want to be spidered in iFrame(s) and disable robots in that file(s).

[Webnauts]

Originally Posted by crankydave

John...

Google is caching the text in the dropdowns so I believe they are seeing it...

Creative Loafing Tampa | Food & Drink | Features | It took a Village

Dave

Oh!!! Then they are more advanced in reading forms as I was aware of. Thanks for sharing Dave.
Well the best and legitimate way to go, would be to move that part of the code below the main content and still have it being seen in the browser where it is now.

Should we go a bit grey or black hat?

As a said above, theoretically you can cloak, but could be that the SE will perceive that a deceiving cloaking method. So I can not give any guarantees.

You can create a folder called i.e forms and then create a file called i.e form.php and in this file you can add the form content.

Then you add in the site template an PHP include to draw in the content of the form.php file.

Then you forbid the bots in your .htaccess to access form.php with a 403. In addition you can add a rule in the robots.txt to disallow the folder forms.

If you think it would be an alternative for you and you do not see any risk, let me know if you need help with setting it up.

I hope I explained properly. By, I have a sort of script for that too.

[Webnauts]

Originally Posted by activeco

Exactly Dave & Kgun.
Put everything what you don't want to be spidered in iFrame(s) and disable robots in that file(s).

Bro, bots cannot read iframes anyway.

[davebarnes]

THIS spider simulator Search Engine Spider Simulator says that the search engines find ALL the pull down lists.

Interesting that Google appears to only see the food-related lists.

[crankydave]

Dave... My *guess* is that they're seeing it all since it's in the source. Also my *guess* is that the cached text is only showing the food and drink because that tab is what is set as the default display for that page.

Here for example...

Creative Loafing Atlanta | Film | Atlanta Film Festival | Life, death and teen angst

Dave

[Webnauts]

Originally Posted by crankydave

Dave... My *guess* is that they're seeing it all since it's in the source. Also my *guess* is that the cached text is only showing the food and drink because that tab is what is set as the default display.

Dave

Yeah! And the rest using display:none. That is more dangerous that the method I mentioned above.
Showing different content to SE and different to users.

[Webnauts]

Dave if you want to go for the solution of Kgun (Ajax) I just found this if you might would like to try: Simple AJAX call to Hide a Form from bots and Search Engines — JavaScript Junkie

[activeco]

Originally Posted by Webnauts

Bro, bots cannot read iframes anyway.

Pardon? Any (i)frame can be followed just as any other hyperlink.
It correctly sees it as a separate page, but in this case one don't want it to be indexed.

[kgun]

Originally Posted by Webnauts

Yeah! And the rest using display:none. That is more dangerous that the method I mentioned above.
Showing different content to SE and different to users.

John whether you block a Bot from the whole site, a page or a part of a page, you show different content to the user and to the bot, so to me it is only meanigful to present the content that are shown to bots and human beings the same.

Google use AJAX technology XMLHttpRequest, iFrames and may be zero frames that used to be (is still?) regarded as cloacking in their webapplications, like Gmail, Google Suggest, Maps etc.

Related:

SEO and frames in a web 2.0 world.

The best for accessibility, usability and consistency may be to divide the page in 2 ore more files / documents, but that was not the question.

P.S.

display:none

I have never used that as I know on my sites, but I use other technologies, though not to block a bot fram a part of a page.

Webapplcations have come here to stay.

[Webnauts]

Originally Posted by kgun

John whether you block a Bot from the whole site, a page or a part of a page, you show different content to the user and to the bot, so to me it is only meanigful to present the content that are shown to bots and human beings the same.

What is with Image Replacement?

[Webnauts]

Originally Posted by activeco

Pardon? Any (i)frame can be followed just as any other hyperlink.
It correctly sees it as a separate page, but in this case one don't want it to be indexed.

You'r right there; as a separate page.

And now how we solve this now?

We can prevent the target of the <iframe> tag from being indexed by using a robots meta tag set to
"noindex" or disallow the page in with the robots.txt file.

Great. I hope Dave likes the idea.

But isn't that cloaking?

[Webnauts]

Cloaking: Serving different content to users than to Googlebot. This is a violation of our webmaster guidelines. If the file that Googlebot sees is not identical to the file that a typical user sees, then you're in a high-risk category. A program such as md5sum or diff can compute a hash to verify that two different files are identical.

Source: Official Google Webmaster Central Blog: How Google defines IP delivery, geolocation, and cloaking

[activeco]

Originally Posted by Webnauts

We can prevent the target of the <iframe> tag from being indexed by using a robots meta tag set to
"noindex" or disallow the page in with the robots.txt file.

But isn't that cloaking?

No. If that's true then every disallowed page would be considered as cloaking, just because you present content to users and nothing to SE.

[Webnauts]

Originally Posted by activeco

No. If that's true than every disallowed page would be considered as a cloaked one, just because you present content to users and nothing to SE.

I still think this requires clarification, because I was using such practices until I heard some stories from Matt Cutts.

Do you mean that if I show users parts of a page and not to search engines is not cloaking? If that is true and you can guarantee that, I will go back to my site and implement my methods again, and get back my good rankings for the terms "seo" and "search engine optimization" which went approx. 60 positions down.

[activeco]

Originally Posted by Webnauts

If that is true and you can guarantee that...

Ha ha, you know best there are no guarantees in this field.
However throwing some non-important pages for a test will not cost you a lot.

[kgun]

Originally Posted by davebarnes

Is it possible to block/prevent search engines from spidering PART of a page?

I know how to seal off part/all of a website. My question here is: if I let a search engine spider get to a page can I separate the page into 2 parts: allowed for spidering and DISallowed?

If yes, how do I accomplish that?

thanks,


dave

1. I repeat this Kjell Bleivik, CV and some poems etc. in Norwegian only link.
2. Look at the source of
   
   - Kjell Bleivik, CV and some poems etc. in Norwegian only
   
   - Kjell Bleivik, leftmenu or frame of the site http://home.no/kjellbleivik/
   
   - Kjell Bleivik. Main menu or frame of the site http://home.no/kjellbleivik/
3. Isn't it possible to use ordinary old frames?
4. And I repeat this SEO and frames in a web 2.0 world. WPW thread.

Why should a bot be interested in a menu aside from following the links that the webmaster means is insemantic (for the bot)?

[activeco]

I remembered and found one old GoogleGuy's (Matt Cutts?) post, three years ago, in another forum. It goes like this:

One thing that is more clear if you read update threads in one big chunk (instead of in-the-trenches as posts are being made) is how folks can get off onto different (and sometimes mistaken) tangents. One of those tangents was that Google would penalize pages that had . I've never heard the suggestion that Google would penalize for iframes before reading it in the thread. Plenty of legit sites use iframes, so it wouldn't make sense to penalize for it. Powdork gave the right response in message 337 of the first Bourbon thread. Now I can easily believe that some search engine spiders would have trouble with iframes just like some spiders have trouble with frames. But I wouldn't expect iframes to cause any penalties. So it's kind of like the whole urban myth that was going on for a while about "if you use javascript to change the text in the status bar, a search engine may penalize you." Sometimes these things just get started, I dunno where. Most of the time there's level heads around to say "that just doesn't sound right; it doesn't hold together with common sense." So I'll debunk the iframe myth now for good measure.

[activeco]

Originally Posted by kgun

- Isn't it possible to use ordinary old frames?

True.

[kgun]

Originally Posted by activeco

I remembered and found one old GoogleGuy's (Matt Cutts?) post, three years ago, in another forum. It goes like this:

• GoogleGuy Definition From The Glossary of Internet Terms - Strategic Web Ventures
• GoogleGuy Says - Google Ranking Info

[Webnauts]

Originally Posted by activeco

Ha ha, you know best there are no guarantees in this field.
However throwing some non-important pages for a test will not cost you a lot.

OK. Here is the test: Hidding content from search engines

[activeco]

Originally Posted by Webnauts

OK. Here is the test:

I failed to see any frames or robots exclusions there. Or do you want to link to it from another place?

[Webnauts]

Originally Posted by activeco

I failed to see any frames or robots exclusions there. Or do you want to link to it from another place?

We created a php script to be able to hide parts of a page content. Wait until it gets indexed, and then check Google's cache.

[activeco]

Originally Posted by Webnauts

We created a php script to be able to hide parts of a page content. Wait until it gets indexed, and then check Google's cache.

I have seen that, but I can't see the connection, unless you created frames only for bots.
Anyway I would suggest starting a new thread as this becomes a bit of hijacking (sorry davebarnes).
Let us know the location.

[Webnauts]

Originally Posted by activeco

I have seen that, but I can't see the connection, unless you created frames only for bots.
Anyway I would suggest starting a new thread as this becomes a bit of hijacking (sorry davebarnes).
Let us know the location.

I just make parts of the page invisible to bots. No frames etc. Pure server side. To be a bit explicit, I start with a spider detection and the game goes on. And I think we are not going off topic at all. Dave wants to hide content from bots. Or did I miss something?

[Webnauts]

Originally Posted by activeco

Let us know the location.

Do you mean location of the script? If yes, http://www.seohackers.net/spider.php

But don't hack my server please...

[activeco]

Originally Posted by Webnauts

I just make parts of the page invisible to bots. No frames etc. Pure server side. To be a bit explicit, I use a spider detection. And I think we are not going off topic at all. Dave wants to hide content from bots. Or did I miss something?

Apparently. A few people here including myself have been advocating (i)frames in achieving hiding. Thus not pure cloaking as your setup does (please don't associate my name with it).

Instead the blue part of your page should be something like this:

<IFRAME SRC="hidden_menu.html">Eventual non-spammy content here such as:Your browser does not support frames. Please visit <A HREF="hidden_menu.html"
TARGET="_blank">Menu</A> in a new window.</IFRAME>

The hidden_menu.html should be robots disallowed, containing the blue part (among other html):

<p><font color="blue"><strong>This page was created as Activeco advised me, and this sentence here (with the blue font color) should not be visible to search engines.</font></strong></p>

[Webnauts]

I repeat two previous posts:

A. Cloaking: Serving different content to users than to Googlebot. This is a violation of our webmaster guidelines. If the file that Googlebot sees is not identical to the file that a typical user sees, then you're in a high-risk category. A program such as md5sum or diff can compute a hash to verify that two different files are identical.

Source: Official Google Webmaster Central Blog: How Google defines IP delivery, geolocation, and cloaking

B. We can prevent the target of the <iframe> tag from being indexed by using a robots meta tag set to "noindex" or disallow the page with the robots.txt file.

-------
My question now is, what is the difference? Dave would like to serve different content to users than Googlebot? If he does it with iframes, ajax or server side scripting, where is the difference? I still don't get it.

[activeco]

The difference is that an (i)frame does not imply cloaking in any way.

[Webnauts]

Originally Posted by activeco

The difference is that an (i)frame does not imply cloaking in any way.

In anyway? And why does my technique imply cloaking? Can you please explain the difference? Because if Google would like to view the content violating any HTTP headers conditionals or robots.txt rules?

Again:

Cloaking: Serving different content to users than to Googlebot. ... A program such as md5sum or diff can compute a hash to verify that two different files are identical.

[activeco]

I felt it will go this way, that's why I suggested a new thread. You're the one doing the test, you should start it.
Anyway, if you insist: Checksum or diff will return no differences when using frames.

[Webnauts]

md5sum or diff

Users see this:



Google see this:


No matter if iframe, javascript, ajax or php. Whats next?

[Webnauts]

Originally Posted by activeco

I felt it will go this way, that's why I suggested a new thread. You're the one doing the test, you should start it.
Anyway, if you insist: Checksum or diff will return no differences when using frames.

And with my server side solution will checksum or diff return a difference? Hey man, we are not off-topic. Dave wants to know how he can hide content in a legitimate way and that is what we are trying to figure out. Or am I still missing something?

[activeco]

Webnauts continued the thread variation here, which does not mean that this thread is definitely answered.

[Peter (IMC)]

Originally Posted by davebarnes

Here is the page in question: Creative Loafing Tampa | Food & Drink | Features | It took a Village

Here is the problem.
I want the search engines to focus on indexing the contents of: the main article and the 3 blocks on the right-hand side as these are all about the restaurant being reviewed and other food and drink.
But, right now, the spiders also see all pull-down lists in the box at the top. These items are detracting from the FOCUS of the article.

In other words, I want the spider to visit and leave with thoughts of "wine and homogenous American fine dining" in its head and not with "Ethiopian, Filipino, Fondue, French" and all the other lists contained in the Ajax construction.

Don't worry about that. It shows up in all pages, right? It's treated differently, similar to the menu of the site. Even though it shows up in the cached pages, I don't see this as a potential problem.

In fact, it could have some benefits even for very long search phrases.

[Webnauts]

Dave and everybody else here, I think I found the solution you might be looking for here:
Google and Cloaking