Censoring the search engines...

[brianzajac]

Hey everyone,

I was in a recent dilemna earlier and one of the members here brought up a good point on censorship. Let's say you wanted to talk about a couple of key points in an article you wrote. However, due to spam issues, you don't want to attract attention to particular keywords. But, you do want people to be attracted to other specific keywords and still create an interesting topic. How can you censor some keywords to an article and promote others in that same article?

[publicgadfly]

TABOO-

(sarcasm on)
Let's not discuss the weaknesses of Outlook, might offend someone
Let's not discuss the 'accursedness' of Spam too heavily or we will offend the opt-in folks,
Let's not discuss hiding or “munging” our e-mail addy because some harvester folks won't like it
Let's only talk about the above board known things so that we can be ill prepared to deal with the other things that will happen. This way we will only offend ourselves.

To fail to acknowledge something dangerous or distasteful is to wallow in stupidity.

[kthulah]

I'm not sure exactly what you mean. If you mean that you want to emphasize some points and not others, then perhaps repitition would help.

Like, instead of writing just the title and article on that page, you could provide the title, a brief summary, and then the article, and then highlighted key sentences between paragraphs. Sort of like what the magazines do to keep people's attention during a long article.

This would mean that certain phrases would be more relevent to a bot than others, because they're being repeated, though not one line after another.

[Mr Fudge]

Have the same problem!
http://www.baby-toys.co.uk/teddy_bear_puppet.htm
:-)
Steve

[minstrel]

Originally Posted by brianzajac

Let's say you wanted to talk about a couple of key points in an article you wrote. However, due to spam issues, you don't want to attract attention to particular keywords. But, you do want people to be attracted to other specific keywords and still create an interesting topic. How can you censor some keywords to an article and promote others in that same article?

I may be wrong but I don't think the question is about "political correctness" - I think brianzajac is asking quite a different question here:

Let's say I have an article on a page on my website. I want the article indexed by the spiders but, to avoid being spammed or harassed any more than necessary, I wish to exclude certain portions of that article, NOT from being read, but from being spidered. This would be similar to using <META name="robots" content="noindex, nofollow"> to exclude a whole page from being spidered, except just a portion of that page.

It's an interesting and legitimate question. I use the AtomZ site search engine on my site and there I can add some code to block out a certain section of content on any page and identify it to AtomZ as content I don't want indexed - there are a number of very valid reasons for doing this. However, outside of that block(s), the page is indexed normally.

If I'm correct, Brian's question is: how do I do this for Googlebot and other spiders?

On the other hand, if I'm incorrect, please feel free to bombard me with derisive catcalls and hoots of laughter, the way my teenage sons do when I make a (very rare) rare mistake :-)

[Whit]

Working on Minstrel's understanding of the question.
I know iFrames are unpopular for several good reasons, but if it had to be done couldn’t you use an iFrame with the borders set to 0 and have the targeted page use the noindex tag? This would give the viewer the illusion of a seamless page while telling crawlers to avoid the parts you want avoided.
This could only be “easily” implemented if you were able to isolate whole paragraphs that you did not want the crawlers to see. Each of these paragraphs would be separated into it’s own page and loaded into a separate iFrame where needed in the context of the main page. It would still be a real pain in the neck though.
There would also be the support issue to consider. While the latest versions of IE Netscape and Opera all support the iFrame, I think a lot of people are still using browsers, or versions of browsers, that don’t.

Just an idea.

[publicgadfly]

I apologize to anonymous and anyone else for my “sarcasm on” post that was in any way offended. I did not intend my post to be taken thus.

I was trying to offer my view that taboos have no place in discussions. For me, I want to know.

I (personally) believe that avoiding something is #@$&*, but that's just me.

If someone has discovered “the golden apple” and wants to share it- I WANT IT!

[stylmedia]

An obvious solution is to create the sensitive subject matter as an embeded sliced image, this would work well if it was a temporary article, not your main content. Or perhaps just use images for the censored words, sized and placed in the correct position. This may not be the best solution, and certainly to me would be more work than it's worth, but if it is important to your audience, then the extra effort is required.

[brianzajac]

Thanks for the info everyone. Actually, Minstrel was right on the money. What I ultimately hope for is a term like <META name="robots" content="noindex, nofollow">, but only for specific words and phrases. Although this code doesn't exist, this is what I ultimately look for:

<META name="robots" keyword content exclude ="bad term 1, bad term 2, bad term 3">

and so on...

I've heard many responses about how to manipulate the page, but that ultimately lowers the importance of the page...and could (in some way) be considered spammy itself. I've also heard of doing text the graphical way, but then again...that's not only very time consuming (imagine making a graphic for every letter and then matching them up to fit with the rest of the text), but adds unnecessary code.

There was an earlier post on a christmas SEO wish, I guess this would be it...hey Google, make me a keyword phrase-based exclusion metatag!

[chris_g]

There is a fairly simple answer to this question that allows you to stop indexing within text and starting indexing again where you want it to restart. Reputable search engines will likely follow these guidelines.

To prevent part of a page from being indexed add the tag

before the section of the page to be ignored, and the tag

after the section of the page to be ignored, then respider your site. The spider, when it notices this tag, will prevent the text occurring between these tags from being included in the index. Note that the spider will still follow the links on this page to locate the other pages of your site.

Inktomi Index Tags
In order for Inktomi to be really effective as a search tool, each page should contain meta tags to allow categorisation of the data. You can also include start and stop tags that tell Inktomi which parts of a page to index. For example the word 'tax' may be repeated on every page as part of the navigation. You would not want every page to be returned if a user searches on 'tax'.
Start Index Tags should appear in your header files AFTER the </head> and <body> tag.

e.g.
</head>
<body>



<START INDEX>

Stop Index Tags should appear in your footers BEFORE any footer navigation.

e.g.

<STOP INDEX>



You can also put Stop and Start tags around any content you wish to exclude from an index of your page i.e. you can have multiple tags on a page.

Inktomi will not index any content between a <STOP INDEX> and a <START INDEX> tag.

[minstrel]

Originally Posted by chris_g

To prevent part of a page from being indexed add the tag

before the section of the page to be ignored, and the tag

after the section of the page to be ignored, then respider your site. The spider, when it notices this tag, will prevent the text occurring between these tags from being included in the index. Note that the spider will still follow the links on this page to locate the other pages of your site.

<START INDEX>
Stop Index Tags should appear in your footers BEFORE any footer navigation.

<STOP INDEX>

Inktomi will not index any content between a <STOP INDEX> and a <START INDEX> tag.

Thanks, chris_g:

I wasn't aware of this but it's very similar to the syntax for my local site search service mentioned earlier.

Question: do you mean to indicate in your post that the syntax for Inktomi is different from the "standard"? i.e., are there TWO formats for doing this? If so, can you use both on the same page?

[brianzajac]

chris_g,

Thanks for the info...that will definitely head me in the right direction. I'm also interested in the question minstrel had...is inktomi different than the rest? And, most importantly, this code works well with Google? Thanks!

[weegillis]

Originally Posted by Mr Fudge

Have the same problem!

-------
"Teddy bear..."

The phrase is repeated, what, 23 times?

_62 words are NOT one of Teddy OR Bear
_21 are Teddy | 23 if you count the Title & DESC
_19 are Bear ..| 21 "
-------------------------
102 Total . . . . |117 "
====================

0.205 density for TEDDY
0.186 density for BEAR
-------------------------
0.391 Total Density

We can easily see your problem here. 40% phrase density definitely looks like spam if you're a robot.

Teddy bear, teddy bear, (repeat phrase)
... touch the ground
... turn around,
... pick up your shoe,
... that will do,
... go upstairs,
... say your prayers,
... blow out the light,
... say goodnight.

This helps to get rid of 14 occurrances of each word.

Teddy drops to- 0.0945
Bear drops to--- 0.0675
--------------------------------
total density----- 0.1620

More within range.

What else can you do? Get the robot to skip the page, and pick up your TEDDY BEAR phrase density somewhere else on the site, I suppose.

This is my stab in the dark...

Roy

[weegillis]

Originally Posted by brianzajac

...hey Google, make me a keyword phrase-based exclusion metatag!

-------
Which would be nice if Google even gave a darn about meta tags.

Probably the best way to exclude text from being indexed is to put it in an image. You could treat the content as 'archive,' photograph a typewritten page, include a standard placement for a 200w x 300h thumbnail, and let your reader pop it open to read it.

Your actual text can be broken into a standard classification header, and a comments or critical writing section where your keywords would be carefully chosen to skirt the 'detritous.'

This would quickly eat up your disk space, but if you have room, and the allowable bandwidth, your readers will probably not mind the extra download.

Roy

[chris_g]

minstrel asked:
"
Question: do you mean to indicate in your post that the syntax for Inktomi is different from the "standard"? i.e., are there TWO formats for doing this? If so, can you use both on the same page?"


The most popular format is <STOP INDEX> and <START INDEX> surronding text you don't want indexed.

I am not aware of an official format that all search engines will follow.

[minstrel]

Thanks, chris_g.

[mrmylesdean]

...then there's always Flash...
if you don't mind "flasheyness" on your site, that is.

/Myles