Google and Cloaking

[Webnauts]

Cloaking: Serving different content to users than to Googlebot. This is a violation of our webmaster guidelines. If the file that Googlebot sees is not identical to the file that a typical user sees, then you're in a high-risk category. A program such as md5sum or diff can compute a hash to verify that two different files are identical.

Source: Official Google Webmaster Central Blog: How Google defines IP delivery, geolocation, and cloaking

Now my question is:

There are some ways you can hide content parts from search engines. For example:

- One option is using IFRAMES. You can prevent the target of the <iframe> tag from being indexed by using a robots meta tag set to "noindex" or disallow the page in with the robots.txt file.

- Another option would be doing that with Ajax like Simple AJAX call to Hide a Form from bots and Search Engines — JavaScript Junkie

- Another option would be You can create a folder called i.e forms and then create a file called i.e form.php and in this file you can add the form content.
Then you add in the site template an PHP include to draw in the content of the form.php file.
Then you forbid the bots in your .htaccess to access form.php with a 403. In addition you can add a rule in the robots.txt to disallow the folder forms.

- Another option is a php script called spider.php we have created and that disallows bots to access content parts where they look for example like this:

PHP Code:

<?php
if (!spider_detection('spider'))
{
echo('<p><img src="http://www.seoworkers.com/images/spam.png" alt="Spam?"></p><p><font color="blue"><strong>"This page was created as Activeco advised me. The page image and this sentence here (with the blue font color) should not be visible to search engines."</strong></font></p>');
}
?>

Are any of the above options violating Google guidelines about cloaking? If yes, which and why?

For the last option you can have a look at a test page I just setup for testing:
Hidding content from search engines

Thanks for your contribution.

John

[activeco]

OK, to continue from here:

Originally Posted by Webnauts

And with my server side solution will checksum or diff return a difference?

Yes, it will. What bots get is not exactly the same page as a regular browser gets and that is a typical cloaking.
Unlike when using (i)frame where the html+content of that page remains exactly the same. For now, content of the linked page is not important in this case.

P.S. Currently your setup returns 403 for active.html, which was probably not what you wanted.

[Webnauts]

Originally Posted by activeco

OK, to continue from here:


Yes, it will. What bots get is not exactly the same page as a regular browser gets and that is a typical cloaking.
Unlike when using (i)frame where the html+content of that page remains exactly the same. For now, content of the linked page is not important in this case.

The content of the linked page is important. That is what I would hide with an IFRAME, disallowing bots to follow the target though a noindex meta tag or robots.txt.

Originally Posted by activeco

P.S. Currently your setup returns 403 for active.html, which was probably not what you wanted.

What is the difference if I would disallow the bots accessing a page with robots.txt or with a 403?

[activeco]

The return codes not the same (except temporary problems) = cloaking.
Html not the same (except everchanging pages) = cloaking
Html the same, the same return code, different rendering due to frames or CSS = not cloaking (yet).

[activeco]

Basically, do not intentionally abuse neither frames nor CSS (or server to that matter) and you're on the safe side.
Don't hide. If a SE is not capable or does not want to follow something, well there you can sometimes improve the perception of your page, but hiding beyond agreed rules will put you at risk.
If a bot wants to see something normally allowed to it, let it see.
If a user comes, show him/her the page as there is no search engine.
E.g. even a CSS hiding could be dangerous: Communication in other languages

[kgun]

Still discussing? Here is another technique: DOM:element.innerHTML - MDC

Related:

Here you will find more about inner and outer HTML

MSDN Library (Click Web developement + HTML and CSS)

You can also search on that page for frames etc.

Example of frames in use: HTML and DHTML Overviews and Tutorials

[Webnauts]

Originally Posted by activeco

The return codes not the same (except temporary problems) = cloaking.
Html not the same (except everchanging pages) = cloaking
Html the same, the same return code, different rendering due to frames or CSS = not cloaking (yet).

OK. Lets try this.

I hide the part of the page content with my script, and redirect the bots with a 301 to another html page where that part of content will be found, and I add there a noindex meta robots or disallow it in the robots.txt. Exactly the same practice I would implement if I would use IFRAMES.

How about that?

[activeco]

Originally Posted by Webnauts

OK. Lets try this.

I hide the part of the page content with my script, and redirect the bots with a 301 to another html page where that part of content will be found, and I add there a noindex meta robots or disallow it in the robots.txt. Exactly the same practice I would implement if I would use IFRAMES.

How about that?

Again, don't hide anything.
I am not sure why do you so desperately insist on different cloaking versions, instead of trying the suggested way. Why don't you simply implement the frames test as intended?

[Webnauts]

Originally Posted by activeco

Again, don't hide anything.
I am not sure why do you so desperately insist on different cloaking versions, instead of trying the suggested way. Why don't you simply implement the frames test as intended?

1. Our sites are build in XHTML Strict 1.0 and this document type does not support iframes.
2. We are seriously concerned about the accessibility of our web sites, and iframes are problematic there.

Any point 1 is enough, or? About accessibility there are some workarounds, but I am not happy with them.

What do you suggest now?

[activeco]

Originally Posted by Webnauts

1. Our sites are build in XHTML Strict 1.0 and this document type does not support iframes.
2. We are seriously concerned about the accessibility of our web sites, and iframes are problematic there.

What do you suggest now?

Trying Transitional instead?
Anyway, the original idea was testing frames for particular purpose and not dealing with XHTML Strict 1.0.
That is another issue.

[Webnauts]

Originally Posted by activeco

Trying Transitional instead?
Anyway, the original idea was testing frames for particular purpose and not dealing with XHTML Strict 1.0.
That is another issue.

Are you telling me I should move backwards? Sorry, but that will never happen. So, since this is another issue, what do you suggest now?

[activeco]

I don't see a big problem. If you want the frame (testing) use Transitional only on pages that need it.
What do you actually don't want to be indexed?

[Webnauts]

I was hiding with our spider script the help navigation for people with disabilities which looked like this:

<body id="www-seoworkers-com">
<?php
if (!spider_detection('spider'))
{
echo('<ul id="skip-link" title="Keyboard Tabbing and Screen Reader users functions">
<li><a href="#main-content">Skip to main content</a></li>
<li><a href="#sub-nav">Skip to sub-navigation</a></li>
</ul>');
}
?>
Then the first visible word in our body was "SEO". At some point we came up for this keyword to #56.
After we took that out we dropped about 60 positions. This is 100% the reason.

So whats next?

P.S. Google indexed the test page. Look at it carefully: site:seohackers.net - Google Search

[activeco]

Hm, personally I wouldn't hide anything.
Maybe playing with CSS to move the menu deeper down in the HTML, while visually maintaining the same position; or even making a full separate page with the menu, which will be robots disallowed?

[activeco]

BTW, what's wrong with XHTML 1.0 Frameset?

[Webnauts]

Originally Posted by activeco

BTW, what's wrong with XHTML 1.0 Frameset?

Are you telling me here to move from XHTML 1.0 Strict to XHTML 1.0 Frameset? I hope that was a joke.

[Webnauts]

Originally Posted by activeco

Hm, personally I wouldn't hide anything.
Maybe playing with CSS to move the menu deeper down in the HTML, while visually maintaining the same position;

That is the problem. It matters where it is in the code! That must be the first part for a screenreader or in our case Lynx will see.

Originally Posted by activeco

or even making a full separate page with the menu, which will be robots disallowed?

How do you want to achieve that?

[activeco]

Originally Posted by Webnauts

That is the problem. It matters where it is in the code! That must be the first part for a screenreader or in our case Lynx will see.

So, as far as I understand, you don't want to be ranked for that part, but you need it it to be at the top of the code?
I am not expert in CSS, but can't you use absolute positioning or so?
I guess this question would be fast answered in this subforum.

[activeco]

Also, check this url: Layout Gala: a collection of 40 CSS layouts based on the same markup and ready for download!

Note that in all 40 layouts html is exactly the same.

[Webnauts]

Originally Posted by activeco

So, as far as I understand, you don't want to be ranked for that part, but you need it it to be at the top of the code?

Exactly.

Originally Posted by activeco

I am not expert in CSS, but can't you use absolute positioning or so?
I guess this question would be fast answered in this subforum.

No need. It has to be where it is in the code. The visual part is not the problem.

[Webnauts]

And you said above:

Originally Posted by Activeco

or even making a full separate page with the menu, which will be robots disallowed?

And I asked you how can I do that? Did you miss that post?

Please explain.

[activeco]

Originally Posted by Webnauts

Please explain.

There is no much to explain.
If you have so much problems with a small menu, make a nofollowed link on top only for screenreader, move appropriate code&content there and additionally disallow the page.

[Webnauts]

Originally Posted by activeco

There is no much to explain.
If you have so much problems with a small menu, make a nofollowed link on top only for screenreader, move appropriate code&content there and additionally disallow the page.

I assume that you were not aware that implementation of the nofollow attribute on our site is a taboo issue. Any other solution?

We don't even use nofollow on all our blogs.

[Webnauts]

OK. Now I came up with another alternative. Does exactly the same as the iframe does.

I created a file called skip.txt and disallowed in the robots.txt, and in that file I added this piece:

<ul id="skip-link" title="Keyboard Tabbing and Screen Reader users functions">
<li><a href="#main-content" accesskey="2">Skip to main content</a></li>
<li><a href="#helpnav" accesskey="3">Skip to local navigation</a></li>
</ul>

See it life here: http://www.seoworkers.com/includes/skip.txt

Then I added in the web page an include something like this:

<?php require('/var/www/path/httpdocs/includes/skip.txt'); ?>

The page is here: SEO Workers Web Design & Search Engine Marketing Consultants

And it works! Am I cloaking?

[activeco]

Originally Posted by Webnauts

I assume that you were not aware that implementation of the nofollow attribute on our site is a taboo issue. Any other solution?

Well, I guess you have to get rid of all those taboo's and use existing solutions as appropriate.
...or do you think cloaking is the best solution because it is beyond the taboo's?

[Webnauts]

Originally Posted by activeco

Well, I guess you have to get rid of all those taboo's and use existing solutions as appropriate.
...or do you think cloaking is the best solution because it is beyond the taboo's?

So is this last option cloaking? I still don't get it man.

[activeco]

Originally Posted by Webnauts

So is this last option cloaking? I still don't get it man.

Me neither. You are principally against Transitional, against Frameset, against nofollow, but it seems you have no problem with cloaking.

The page is here: SEO Workers Web Design & Search Engine Marketing Consultants And it works! Am I cloaking?

Yes, you are. Additionally, it seems that you forbid the temp.html to all bots and not only the skip.txt file.

[Webnauts]

Originally Posted by activeco

Me neither. You are principally against Transitional, against Frameset, against nofollow, but it seems you have no problem with cloaking. Yes, you are. Additionally, it seems that you forbid the temp.html to all bots and not only the skip.txt file.

LOL man!!!
I disallowed the temp.html temporally as I do not want that Google picks up this test page.
So we said that we can disallow Google accessing the target page of an iframe. Why can't I disallow the target file skip.txt?

[activeco]

Originally Posted by Webnauts

So we said that we can disallow Google accessing the target page of an iframe. Why can't I disallow the target file skip.txt?

Because there will be a difference between the file you show to the SE and the one you show to other visitors.
With a frame your basic file remains the same all the time, even if you change the content of the linked frame.

[activeco]

P.S. As a matter of fact you don't even need to forbid the framed file as its content will not be strongly associated with the linking file.

[Webnauts]

Originally Posted by activeco

Because there will be a difference between the file you show to the SE and the one you show to other visitors.
With a frame your basic file remains the same all the time, even if you change the content of the linked frame.

Can you explain where you see the difference?

Hey and by the way, I think we have to invite Janeth here.

[Webnauts]

Originally Posted by activeco

P.S. As a matter of fact you don't even need to forbid the framed file as its content will not be strongly associated with the linking file.

So it will be hidden from the SE?

[activeco]

Originally Posted by Webnauts

So it will be hidden from the SE?

No. No need, unless you really want to. The linked page will not be considered as a hard part of the linking page.

[activeco]

Originally Posted by Webnauts

Can you explain where you see the difference?

Try checksum for both files.

Hey and by the way, I think we have to invite Janeth here.

OK. Your place or my place?

[Webnauts]

Originally Posted by activeco

No. No need, unless you really want to. The linked page will not be considered as a hard part of the linking page.

I am taking for example the case of Dave Barnes on the other thread. The navigation needs to be hidden.
So, why is the iframe ok and not the included txt file?

[Webnauts]

Originally Posted by activeco

Try checksum for both files.

Well md5 sums won't work, because pages have dynamic content that could change every time. Or?

Originally Posted by activeco

OK. Your place or my place?

You got the choice bro.

[spiderbait]

Originally Posted by Webnauts

OK. Now I came up with another alternative. Does exactly the same as the iframe does.

I created a file called skip.txt and disallowed in the robots.txt, and in that file I added this piece:

<ul id="skip-link" title="Keyboard Tabbing and Screen Reader users functions">
<li><a href="#main-content" accesskey="2">Skip to main content</a></li>
<li><a href="#helpnav" accesskey="3">Skip to local navigation</a></li>
</ul>

See it life here: http://www.seoworkers.com/includes/skip.txt

Then I added in the web page an include something like this:

<?php require('/var/www/path/httpdocs/includes/skip.txt'); ?>

The page is here: SEO Workers Web Design & Search Engine Marketing Consultants

And it works! Am I cloaking?

Maybe I'm missing something, but I don't see how this is cloaking in any way. In fact, I don't see how this modifies the HTML source code any differently for users or bots.

The source code looks the same to me either way.

Did I miss something?

[Webnauts]

Originally Posted by spiderbait

Maybe I'm missing something, but I don't see how this is cloaking in any way. In fact, I don't see how this modifies the HTML source code any differently for users or bots.

The source code looks the same to me either way.

Did I miss something?

We disallow the spiders to access the file spider.txt and its content which is:

<ul id="skip-link" title="Keyboard Tabbing and Screen Reader users functions">
<li><a href="#main-content" accesskey="2">Skip to main content</a></li>
<li><a href="#helpnav" accesskey="3">Skip to local navigation</a></li>
</ul>

What do you think?

[spiderbait]

Originally Posted by Webnauts

We disallow the spiders to access the file spider.txt and its content which is:

<ul id="skip-link" title="Keyboard Tabbing and Screen Reader users functions">
<li><a href="#main-content" accesskey="2">Skip to main content</a></li>
<li><a href="#helpnav" accesskey="3">Skip to local navigation</a></li>
</ul>

What do you think?

I see that, but you're including the contents of skip.txt into temp.html, so the spiders don't need to visit skip.txt and your robots.txt disallowal doesn't make any difference.

Either way, you've use a php require command and every visitor (human or bot) is going to see the same content. So, I don't see how this is cloaking.

[activeco]

Originally Posted by spiderbait

Either way, you've use a php require command and every visitor (human or bot) is going to see the same content. So, I don't see how this is cloaking.

As I understand he's checking for bots too.

[spiderbait]

Originally Posted by activeco

As I understand he's checking for bots too.

I know he was with the previous examples, but he always showed "how" he was checking for bots. Since he didn't with this one I wondered if he'd overlooked something.

Plus, when I spoofed the useragent to check it out, I saw the same content, so his detection must not be working for this example.

On the other example (spider.php) I could definitely see different content depending on useragent.

[Orion]

Is there a problem using the robot.txt file?

[Webnauts]

Originally Posted by spiderbait

Maybe I'm missing something, but I don't see how this is cloaking in any way. In fact, I don't see how this modifies the HTML source code any differently for users or bots.

The source code looks the same to me either way.

Did I miss something?

Damn! I was confused. You are right. That is an include.

Please look at the last option I mention in my original post. That is what I need to know.

[Webnauts]

Originally Posted by spiderbait

On the other example (spider.php) I could definitely see different content depending on useragent.

Yeah. That is the one I want to know what you think. Would you call that technique violation? I show users more content and the SE less. Where is the violation? There is not trick like stuffing or hiding keywords from users and just showing them to the SE.

[Webnauts]

OK guys. Here are the results of the experiement:

The test page Hidding content from search engines was indexed here site:seohackers.net - Google Search and compare the test page with the cache Hidding content from search engines and tell me if you see the difference.

[activeco]

Congratulations. You are at the first position (in two days) for the phrase "Hidding content from search engines" (without quotes), out of 224000 results. Cloaking does pay, at least in the begin.
Let's see how long you will stay there.

[Webnauts]

Originally Posted by activeco

Congratulations. You are at the first position (in two days) for the phrase "Hidding content from search engines" (without quotes), out of 224000 results. Cloaking does pay, at least in the begin.
Let's see how long you will stay there.

I was using that technique on my site for almost two years and nothing ever happen.
But since Matt brought up this topic recently, I removed that practice.

The only thing that I am wondering, why did he ignore my question is his blog a while ago?

I just asked if we can hide navigation parts which are exclusively for blind users. Like Skip links.

[activeco]

Try to add that menu to the test page and see if it will have big influence.

[Webnauts]

Originally Posted by activeco

Try to add that menu to the test page and see if it will have big influence.

Do you mean that I should unhidde the text in blue and hide the navigation we are talking about?

[Webnauts]

I added this as you have requested. Should I still unhide the other text in blue? Or let it there hidden?

<?php
if (!spider_detection('spider'))
{
echo('<ul id="skip-link" title="Keyboard Tabbing and Screen Reader users functions">
<li><a href="#main-content" accesskey="2">Skip to main content</a></li>
<li><a href="#sub-nav" accesskey="3">Skip to local navigation</a></li>
</ul>');
}
?>