Keyword Density

[DanThies]

MJ, I gotcha.

How they might filter out keyword stuffing and other types of spam is a whole other question. Density, as such, isn't mentioned in any research on that though. One of my colleagues went to SIGIR last month and is a lot more in touch with IR research, so I'll ask him next week when I see him.

Dave, I don't care who deleted my post, and I'm going to let the rest of it go unless you want to press the issue.

[crankydave]

Originally Posted by DanThies

Dave, I don't care who deleted my post, and I'm going to let the rest of it go unless you want to press the issue.

You level a completely unsubstantiated accusation at me for deleting a post.

You're welcome to let the issue go or provide any kind of proof for it. Your choice.

If you have an issue with the way this forum is moderated, my suggestion to you would be to email or PM a forum administrator or iEntry about it. Since you posted that you have had a lond standing relationship with iEntry I'm assuming you know who to contact. If not, I'd be happy to provide my suggestions.

Dave

[DanThies]

Well, Dave... I don't know who deleted my post, but I do know who accused me of spamming, and that was you. Given that you seem to be standing by that, I probably will have a chat with someone.

[crankydave]

Originally Posted by DanThies

Well, Dave... I don't know who deleted my post, but I do know who accused me of spamming, and that was you. Given that you seem to be standing by that, I probably will have a chat with someone.

You don't know who deleted your post but you level a completely unsubstantiated accusation as to who did and demand a retraction and apology. Nice.

You want to discuss my original opinions and observations on your original post in this thread I'm happy to. I don't think hijacking this thread and the OP's legitmate question is appropriate for that purpose. Do you?

Dave

[mjtaylor]

Originally Posted by DanThies

MJ, I gotcha.

How they might filter out keyword stuffing and other types of spam is a whole other question. Density, as such, isn't mentioned in any research on that though. One of my colleagues went to SIGIR last month and is a lot more in touch with IR research, so I'll ask him next week when I see him.

Nice to be heard. Yes, my whole point here was that 'excessive keyword density' would just be another way to say 'keyword stuffing' or 'overuse of keywords.'

So, for example, if I had a site about widgets and my title was <title>Widgets Red Widgets Blue Widgets Green Widgets More Widgets</title> that would be keyword stuffing or, put another way, the KW density of the title would be high.

I f I continue to do that throughout the page ... with headings and alt tags and image file names and text that is very excessive in the use of the word 'widget,' I am going to have a problem getting that page to rank well. SEs will see the site as spammy.

I will be interested to hear what your IR friend says.

Thanks, MJ

[Webnauts]

I do not think things are clear here.

Keyword density alone is not the issue. Keyword density + keyword prominence must be your concern.

Just my ... you know what.

[crankydave]

Keyword density, unto itself, really tells you nothing.

In your title example MJ...

Yes, using the same keyword over and over could cause a problem. And yes because of it, the density of that title will be would be too high. But is the density the metric or is it because of the proximity of those key words to each other the metric, or something totally different?

Onother example that is often used is to look at the top 10 sites for a phrase, use a tool or various tools to determine the KW density, and try to work within those parameters or toss out the high and low to figure an average and work with that.

Problem here is at least 2 fold...

If you compute the KW density for sites that rank 1-10, then 11-20, 12-30 etc. etc. etc., you're likely going to find the same commonality for density for each separate list. In other words, again, it tells you nothing.

Additionally, it can't be known whether or not each instance of the phrase being found by the various "tools" is having any affect, positive, negative, or neutral, on the ranking results. When you're using data that can't be determined as a "factor" the conclusion that data draws cannot be accurately used as a factor either. The most glaring examples I can think of would be keyword metas, or comment tags.

When you're using a "solution" as a ranking metric that was arrived at by using data whose effect is not known, not an influence and variable when it is, then using that "solution" as a metric is a mistake IMO.

Dave

[erikko]

how can i achieve the best keyword density? do you have any tool that generates the right keyword for a site?

[crankydave]

Originally Posted by erikko

how can i achieve the best keyword density?

By simply writing naturally.

Dave

[Webnauts]

Keyword weight is calculated as the number of words in the key phrase multiplied by frequency and divided by the total number of words (including the keyword).

Example: The title of a Web page is 'Get Best SEO Services'. Keyword weight for 'SEO Services' is 2*1/4*100%=50%.
If you reduce the number of words in the title by removing the word 'Get', so the title becomes 'Best SEO Services', than the keyword weight will be larger: 2*1/3*100%=67%.
Finally, if you only keep 'SEO Services' in the title, the keyword weight will become 100% -- 2*1/2*100%.

So, to increase the keyword weight, you should either add some more keywords or reduce the number of words in the page area. The proportion of the keywords to all words will become larger, so will the keyword weight.

I hope this makes sense.

[Webnauts]

Originally Posted by crankydave

By simply writing naturally.

Dave

Best tip than any other.

[crankydave]

Here's at least part of the problem in calculating KW density...

Scenerio... You have a page with 500 words. A 2 word phrase appears on that page exactly 4 times...

Once in the meta description, once in the meta keywords, once as an anchor link in the navigation footer, once as the title tag for that footer nav link.

What is the density?

1. As that exact phrase relates to 2 word phrases on the page? 1.6%
2. As that exact phrase relates to all the words on the page? .8 %

Additionally, using the places where I noted the KW's were found, how does the KW density relate to rankings? What does the KW density in this instance tell you?

Dave

[wige]

I have been trying to follow this as best as I can, but most of this is a bit beyond me. Please pardon my newbie-level questions, but I am trying to grasp this without actually seeing the inside of a SE. For example, (please try to suppress laughter) what is IR?

It seems to me that most of this discussion centers around how the keyword density affects placement in the Google SERPs, does this hold true in other search engines? Yahoo and MSN seem to be more influenced by on page factors, where Google seems to be most affected by off page elements such as links and anchor text.

Also, in trying to follow this discussion, I tried to relate the comments to databases that I have worked with; although these databases are much simpler than what the SEs use, they have many similar operators.

For example, my main web site exists mostly as database entries. When a search engines crawls my site, it is adding my content to it's own database. My internal search thus works similar to the search engines, but in a much much more simplistic manner, as far as I know. My database has a field containing the title for the page, another field for the page content, and fields for the description and keyword meta tags. I imagine the SEs do something similar. When I do an internal search based on a keyword query, the results are based on the keyword index, where keyword density is the primary consideration - actually, the only consideration. The first results are the ones where my query phrase exists the most times, the last result is the item that contains one occurrence of one word in my query phrase.

My understanding was that search engines in the early days used a very similar system, but people quickly figured this out, and started exploiting this by keyword stuffing, so the search engines somehow normalized the databases to penalize pages that had just the keywords repeated over and over again.

Not talking about Google, but referring to search engines that focus on internal aspects of the page, what is it that they will rely on to rank the pages when retrieving them from the database than how many times the query string appears in the text and in what form (interceding words, synonyms, containing tags etc) if not the density of the keywords relative to the user's query? And what do they do then to combat keyword stuffing?

[mjtaylor]

Originally Posted by wige

I have been trying to follow this as best as I can, but most of this is a bit beyond me. Please pardon my newbie-level questions, but I am trying to grasp this without actually seeing the inside of a SE. For example, (please try to suppress laughter) what is IR?
<snip>

Not talking about Google, but referring to search engines that focus on internal aspects of the page, what is it that they will rely on to rank the pages when retrieving them from the database than how many times the query string appears in the text and in what form (interceding words, synonyms, containing tags etc) if not the density of the keywords relative to the user's query? And what do they do then to combat keyword stuffing?

IR is Information Retreival ... and the article by E Garcia that Dan referred to is very informative ... though your final question remains valid and unanswered. How so SEs prevent keyword stuffing? And isn't that a density question?

So it seems to me.
MJ

[DanThies]

Originally Posted by crankydave

Once in the meta description, once in the meta keywords, once as an anchor link in the navigation footer, once as the title tag for that footer nav link.

Nobody but Yahoo indexes the meta keywords, they only use it as a last resort, nobody's using the meta description for ranking, and the title attribute of IMG and A tags isn't indexed at all by anyone. So it sounds like you have only one occurence of the phrase, in a link at the bottom of the page.

"Just write naturally" is definitely a much simpler formula.

[crankydave]

Originally Posted by DanThies

Nobody but Yahoo indexes the meta keywords, they only use it as a last resort, nobody's using the meta description for ranking, and the title attribute of IMG and A tags isn't indexed at all by anyone. So it sounds like you have only one occurence of the phrase, in a link at the bottom of the page.

"Just write naturally" is definitely a much simpler formula.

Except that Yahoo will rank a page that has a key phrase that appears only in the meta description and nowhere else.

And yes, I agree with you and everyone else who had already pointed out that "writing naturally" is the best tool you can us.

Dave

[wige]

Well, I have reread the referenced document by E Garcia, a few times, and I am still confused as to how this means that keyword density is irrelevant. This may be partially due to my aversion to math, but my understanding of the process described in the paper is as follows. (For simplicity, this is not considering weighting based on rarity in the database or on-page factors such as different tags or any off-page factors)

The page is retrieved, and the text is stripped, by removing the html tags. The text is then tokenized, where all the punctuation and special characters are removed, leaving only raw text. The text is then filtered, removing the stop words and some language constructs. Next, the text is stemmed, removing plurals, tenses and other such constructs. Finally, a list of the number of times each stemmed term occurs is generated. Most likely a few versions of the text are retained: the list of stemmed words, the stemmed form, the filtered form, the tokenized form and a form with limited markup. Additionally, a search query is probably put through the same steps.

• Original text: Acme Widget Corp sells red widgets, blue widgets and green widgets. Whatever your widget needs, we sell the widget you are looking for.
• Tokenized text: acme widget corp sells red widgets blue widgets and green widgets whatever your widget needs we sell the widget you are looking for
• Filtered text: acme widget corp sells red widgets blue widgets green widgets whatever your widget needs sell widget looking
• Stemmed text: acme widget corp sell red widget blue widget green widget whatever you widget need sell widget look
• Word list: acme (1); widget (6); corp (1); red (1); sell (2) etc...
• Original/Tokenized/Filtered search: Sells blue widgets
• Stemmed search: sell blue widget

So, how does a search engine determine which documents to show for a given query, and in what order? It probably first applies a filter: it looks at the stems that are in the query, and grabs only the 10,000 pages that contain the most occurrences of all, or at least some, of those stems, calculated recursively. The search engine then goes through those 10,000 pages, first looking at the tokenized form of the page vs the tokenized form of the query looking for exact, then close matches, with the pages with the most exact matches followed by those with the most close matches. The search engine then compares stemmed results and so on.

This suggests to me that there must be some minimum number of exact or close matches for a given query in order to be competitive. If you know the exact search term users will enter, this method seems to indicate you would perform better if you have that search term in your text more times than your competitors, thus you have a minimum keyword density goal (keyword density probably being calculated to take into account stem forms etc.).

Of course, if this were the rule, it would be easy to abuse. In fact, this is the basic formula that most search engines 10 years ago used, and it was frequently abused. You would simply create a different landing page optimized for each expected or desired search query, and each time a competitor's page outranked yours you would add a few more paragraphs packed with that phrase. (Worked in the early versions of Yahoo and Altavista, Google didn't exist yet.) So the search engines caught on, and most likely they now just look at the page and say something like "Widget is mentioned 50 times, and there are 200 words on the page, apply spam penalty", so now there is some sort of upper limit to the number of times your keyword or target phrase can appear in order to show in the results.

The document suggests other elements as well, such as weighting, that impact the effect of keyword density on the basis that the weighting factor is invisible outside of the SEs engineering department, but changes the equations to the point that density calculations no longer are relevant. The theory seems to be that those terms appearing rarely on the web as a whole gain additional wieght in your text if you use them. But if that is part of a competitive search phrase, you are using it, and your competitor is using it, so it would no longer matter that the word carries extra weight, and it returns to being a matter of which page uses the term more.

[crankydave]

wige...

In order for any density figure to be relevant, a metric, one would have to know weighting. Yahoo for example, weights words in the meta description for ranking purposes, Google does not. The title of the page is weighted more heavily for ranking. Repeated text link can be normalized.

It is because words are weighted, and that weighting is variable, that unless you knew what the SE's weighed, and how much, and when, any density percentage you come up with really tells you nothing.

As an extreme example, if all the keyword density for a page came from the meta description and meta keywords tags, that page would not rank at all in Google. The density would be irrelevant.

Dave

[wige]

Originally Posted by crankydave

As an extreme example, if all the keyword density for a page came from the meta description and meta keywords tags, that page would not rank at all in Google. The density would be irrelevant.

Do you mean this as is, or do you mean if that page had no meta tags it would not rank because there was no factor with which to weight the terms?

What I took from the referenced page was that the weighting was based upon how common a selected term was in the overall index, which would indicate to me that the weighting for each term would be applied to every page equally. But this would then cancel out the weighting because each competitor uses those words, and it again comes down to the relative density on each page.

If on the other hand, there is an on-page factor such as a meta tag that determines weighting, wouldn't someone have determined this by now?

If weighting on the page is determined based on the text itself, then again the frequency of the term plays into the weighting and is once again relevant, because you need to figure out how frequently those keywords appear in the page relative to other phrases.

Again, most of this is based on old school SEM techniques. I have a dozen binders in my attic full of black hat techniques designed to force pages into the search engines dating back to my early days on the web, and most of the topics involved manipulating keywords and keyword densities. Is most of this relevant today? No, I don't think so. But should this be ignored completely? No. Because if you fill the page with your keyword phrase and drown out the remaining text, the bots will scream spam. And if the rest of your text drowns out your keywords, the bots could miss them. If you are checking your page about widgets, but you mention sprockets twice as often, your page may not do as well for widgets as it could, and by totally ignoring the density you might not even catch this. This would, of course, be more significant on the engines that don't focus on off-page factors, but every little bit is part of the larger equation, and the only way to unravel the mysteries of ranking without working for an SE is to compare these factors and see how they affect the rankings of various pages.

[crankydave]

I mean it as is wige.

Since Google does not weight the meta description or meta keywords for ranking, any page that derived all its density from those 2 place for a particular word or phrase would not rank at all, therefore, any density figure would be irrelevant.

What I took from the referenced page was that the weighting was based upon how common a selected term was in the overall index, which would indicate to me that the weighting for each term would be applied to every page equally. But this would then cancel out the weighting because each competitor uses those words, and it again comes down to the relative density on each page

Okay, Page A and Page B

Consider offpage factors for both pages to be exactly the same. Both pages use exactly the same number of words.

Page A uses a keyword once. Page B uses the exact same keyword 3 times.

Which page ranks higher?

Dave

[wige]

Originally Posted by crankydave

Since Google does not weight the meta description or meta keywords for ranking, any page that derived all its density from those 2 place for a particular word or phrase would not rank at all, therefore, any density figure would be irrelevant.

Got it.

Originally Posted by crankydave

Okay, Page A and Page B

Consider offpage factors for both pages to be exactly the same. Both pages use exactly the same number of words.

Page A uses a keyword once. Page B uses the exact same keyword 3 times.

Which page ranks higher?

I would expect page B to rank higher, but what if each page only has ten words on it? All of a sudden, page B could run into a spam filter and be outranked by page A. But the owner of page B could have prevented this - this high number of keywords to total text would have been discovered by checking the keyword density. On the same token, the owner of page A could have seen that he only uses his key term once, and that he should add to it. Especially if unrelated terms have a higher density.

But also, to expand this example slightly, lets say both pages mention the keyword three times, but page A has a third the total amount of text as page B, with all other factors being the same? Now which ranks higher?

[crankydave]

Originally Posted by wige

Got it.

I would expect page B to rank higher,...<snip rest of qoute>

What if I told you Page B only used the keyword in the meta description and meta keywords tags?

Dave

[DanThies]

Should we assume that the keywords never appeared in links pointing to the page, Dave?

[crankydave]

Originally Posted by DanThies

Should we assume that the keywords never appeared in links pointing to the page, Dave?

Yes.

Dave

[Synonymizer]

I took the trouble to analyze the Keyword Density in metatags, body, domain, subdomain and page name; in sets of pages ranked 1-100, 300-400, 800-900 and others; under different keywords, languages and googles. The results favour different levels of optimal keyword density in different parts of the URL and page. I would be very careful while optimizing, because it is easy to go over the limit and get penalized. The raw data and the tool are available for those who ask them.

[crankydave]

Originally Posted by Synonymizer

I took the trouble to analyze the Keyword Density in metatags, body, domain, subdomain and page name; in sets of pages ranked 1-100, 300-400, 800-900 and others; under different keywords, languages and googles. The results favour different levels of optimal keyword density in different parts of the URL and page. I would be very careful while optimizing, because it is easy to go over the limit and get penalized. The raw data and the tool are available for those who ask them.

I would be interested. Please feel free to PM me when you have a moment.

Dave

[dharrison]

Originally Posted by incrediblehelp

Ignore "keyword density" and write for you visitors and your topic.

I agree and this point was made more than once at SES London.

By all means mention your key phrase in the page title, heading and hopefully the body of the text, but don't get too into the whole density thing and what percentage.

If it doesn't sound natural to your target audience, they're going to see right through it and even find a site that talks to them rather than the search engines.

Hope this helps.

[Webnauts]

Originally Posted by dharrison

I agree and this point was made more than once at SES London.

By all means mention your key phrase in the page title, heading and hopefully the body of the text, but don't get too into the whole density thing and what percentage.

If it doesn't sound natural to your target audience, they're going to see right through it and even find a site that talks to them rather than the search engines.

Hope this helps.

I fully agree with you Debbie. But reading MJs post here One way of dealing with Google I am sure she will not.

I still posted some kind of on topic info on another thread this morning: Can You Combine Writing Naturally and SEO Copywriting?

[crankydave]

That's enough John.

Thank you

Dave

[Webnauts]

Originally Posted by crankydave

That's enough John.

Thank you

Dave

You are right Dave.