Formula For Keyword Connectivity

[Garrett]

The Search Engine Watch forum is off to a grand start with a hyper-scientific keyword research post (http://forums.searchenginewatch.com/forum/showthread.php?threadid=48) from Orion, "a formal scientist, with special interest in AI applied to IR technology." His posts focus on "keywords semantic connectivity and what it can do for improving success across search engines."

The gist of his post, as I understand it, and I look forward to other interpretations, is that in order to find the best traffic for your keywords you should use semantically-related keywords on your site.

I derived this from a statement farther down the thread: "thus in Google, k1=car and k2=automobile seem to have a greater synonymity association (semantic connectivity) than k1=car and k2=auto." The word "auto," as Orion says, occurs in languages other than English - it's also the root of several other English words.

It's interesting to note however that when he ran his analysis on "car insurance" vs. "auto insurance" that "auto insurance" actually had more connectivity.

So how does his formula work? Here's his direct quote:

"Let n1 and n2 be the number of search results containing k1 and k2, respectively and n12 is the number of search results containing both terms. (One actually does a search for k1 then for k2 and finally a composite query consisting of k1 and k2). Using geometry arguments and fuzzy sets, it can be demonstrated that there exists an index, termed correlation index, c, such that

c = n12/(n1 + n2 - n12)"

This formula apparently helps most in comparing the connectivity of keywords - a higher connectivity helps with higher rankings. Why? Well, let's dig into the formula and see what happens.

(Orion emphasized that the keywords you compare have to be related, but not the same. I guess another word for that would be "synonyms.")

I'm using Google.

k1: dog
k2: canine

Here are my results:

n1: 52,700,000
n2: 1,860,000
n12: 999,000

c=999,000/(52,700,000 + 1,860,000 - 999,000)
c=0.0186516309

That's great, but now we need to have something to compare, and, as I understand it, this formula is for finding the keywords that will have the most connectivity and therefore relevance. So now we'll substitute in a keyword there for k2 - "pooch."

Our new computation, with the results for pooch put in there, is
c=999,000/(52,700,000 + 264,000 - 999,000)
c=0.019224478

Now, if I performed everything properly then from these numbers we see that the combination of dog and pooch have a slightly higher connectivity than dog and canine. Orion states that "the optimum combination of k1 and k2 is that one with the highest c-index."

Honestly I'm not sure why that would be - it seems that since there are more instances of the word "canine" than "pooch" that there would naturally be more connectivity between "canine" and "dog." That's apparently not the case though.

Orion pointed out that you should, "repeat recipe for other search engines. c-indices may change, which indicates that semantic connectivity is different across databases. (Very important for SEOs!)"

I'm not sure all you seo copywriters out there should necessarily get your calculators out as you write. I've yet to hear any trusted SEOs weigh in on the subject yet, and this is the first I've heard of Orion.

Read his post (http://forums.searchenginewatch.com/forum/showthread.php?threadid=48) and tell me what you think.

[Ed Stockdale]

tell me what you think
I think Orion needs more bran in his diet. To quote another "formal scientist" he's spoofing you off. Perhaps he will also give us a fromulea for picking lottery numbers:=)

[Mel]

Its all very well to pronounce complicated theories that rely on long calculations to improve your SEO ---> So long as there is some evidence that search engines actually use these techniques in their algorithms.

IF they do then its fine to try to understand them, if they do not then you may be going 180 degrees from where you want to go.

First things first IMO, show that there is some relevance of this theory to actual search engine use.

[Dave Hawley]

"keywords semantic connectivity and what it can do for improving success across search engines."


Noun 1. connectivity - the property of being connected or the degree to which something has connections

Adj. 1. semantic - of or relating to the study of meaning and changes of meaning; "semantic analysis"

Now, perhaps someone can re-word the statement into laymen's terms?

What is his definition of "improving success across search engines" more traffic?

Quite frankly it's sounds like all talk no action from someone who has maybe never run a online business let alone a succesful one.

[moosedsm]

i thing you're working under a content level error. would'nt dog relate more to pooch than canine? canine is a sicentific term not normally used in everyday language, while dog is more commonly used. the average searcher is goining to lookup dog and pooch before trying canine. i thing this is an area that needs to be thought-out more by web authors. top level words are nice, but most people don't use them.

[rdroske]

It appears that your calculation is incorrect. The value for n12 (the number of results for both terms) is the same in both cases and I am sure that is not right.

[Peter (IMC)]

It seems to me that understanding what he wants to show is the first problem. In his article he says:

According to Fuzzy Set Theory ("Modern Information Retrieval"; Baeza-Yates; Ribeiro-Neto, Addison, 1999), the degree of term co-ocurrence in a database is a measure of semantic connectivity (SM) and can be used to build thesaurus for the database. Some engines use term co-occurence in their query expansion algorithms. Understanding how one can measure term co-occurence could be used to carefully select keywords semantically connected in a given search engine database. As an added benefit, SM makes unnecessary the excesive repetition of keywords (keywords spamming).
Now what does that mean?
Simply said: Look at how often (at least) 2 words occur together and compare that to how often these words occur on their own. The more often the words occur together compared to the total occurances of the words, the more semantically connected they are.

The formula Orion made simply calculates a percentage. How often 2 words occur together out of all pages that mention at least 1 of the words.

So what you calculate is which phrase has a higher relative density in the index. Relative because absolute numbers are canceled out in this formula. So it doesn't say anything about how many pages contain the words.

This is all nice, but for what can we use this new knowledge? Orion says:

a. test the best combination of paired keywords from a pool of keywords with the highest semantic connectivity (for that database).Does it help in SEO? I doubt it. We want to know how often a phrase is searched for.


b. build a thesaurus of synonisms targeting that database.That could work, but you still would have to come up with the words your self.


c. build a query expansion or find similars library.I don't understand exactly what he means here. Sounds like it is the same as in b.


d. carefully craft titles and descriptions of web pages.Perhaps, if you are really bad at writing, this could be useful.

It is my impression that these types of theories are of more use for ranking than they are for SEO. I would think that that phrases with a high semantic index will result in favoring pages that contain the complete phrase. That´s pretty much what we´re seeing already and what we would prefer in SEO.

Regards,

Peter

[jerrymac]

This is Jerry Mac with The Country Music Planet located on the Internet at http://countrymusicplanet.com. The article titled "Keyword Formula May Offer Ranking Advantage" is totally rediciculous. I speak with authority as someone who has maintained excellent search engine positioning in most of the major search engines. The key phrases that I have targeted are always real high. Here are a few examples, only a few of many. We are so easy to find in the search engines that all you have to do is go searching for anything anywhere relating to what we do. As of June 8, 2004, at Yahoo.com searching for country music we are #18 out of 38,400,000 results; at Google.com searching for country music singers we are #3 out of 734,000 results; at AOLSearch.com searching for country music songwriters we are #2; At AltaVista.com searching for country music singer we are #2 out of 914,399 results;at MSN.com we are #1 searching for country music entertainers. These rankings have all been achieved by yours truely without any help from anyone other than information I have discovered in news articles in publications like this one. We have never paid one red cent for a listing in a search engine. The one who wrote this article needs to continue writing articles, because he would not make a very good person to be attempting to get search engine placement. I only state the above facts about how well I have done in the search engines to give some credibility to what I am saying. Please, don't place any more articles this rediculous in your publication. It does not belong in the company of so many informative and helpful articles I have discovered here. Thank you Jerry Mac jerrymac@countrymusicplanet.com http://countrymusicplanet.com and http://countrymusicplanet.net.

[cbp]

jerrymac --

You are way off base with your comments. The example of your site doing well has nothing even remotely to do with showing that the article is ridiculous.

...that does not mean I agree with the article, its just your arguments are flawed (or non-existent)

CBP

[jerrymac]

I disagree with the fact that my comments were way off base. I said in the comments that the only reason I was mentioning how good my sites were doing is to give credibility to what I was saying about the article. I realize that it has nothing to do with how good or bad that my site is doing, but the fact that my site is doing good makes me an expert in the field of search engine positioning, and I challenge anyone to do better than I can with the same tools I am working with including you. I stand by my original comments that this article does not belong in the company of the other great articles I have seen posted here. Don't want to discuss this any further, but you feel free to discuss away all you want to.

[cbp]

All my sites rank well for a lot of terms too and I never paid a cent to anyone either - whats that got to do with the article being ridiculous?


Don't want to discuss this any further

Another one hit wonder who claims something is ridiculous but refuses to back it up.

CBP

[Peter (IMC)]

Jerrymac,

For someone saying that that article is ridiculous you certainly have a very ridiculous way of self-promotion going on in your post.

That article is not ridiculous. It actually is very interesting and the theory he uses is not ridiculous either.

My opinion on what he described you can find in an above post.

Oh by the way,.. if you do a search in Google for "country music singers" (with the quotes) in google you will find something like 3700 sites competing. I´m sorry,.. but that´s not impressive.

Just be respectful please. Perhaps you don't like discussions about these types of theories, but then again,.. not everybody likes country music either.

Best regards,

Peter

[giftsun]

Jerrymac says "I disagree with the fact that my comments are off base". How can he disagree yet agree that it is a fact? If you disagree it means you don't think it is a fact.
I just found this response to be somewhat on the comical side.

Herman
www.giftsunusual.com

[Dave Hawley]

Well you guys are doing better than me, I still cannot make sense of the Title



Quote:
"keywords semantic connectivity and what it can do for improving success across search engines."



Noun 1. connectivity - the property of being connected or the degree to which something has connections

Adj. 1. semantic - of or relating to the study of meaning and changes of meaning; "semantic analysis"

[Mel]

I suppose there must be lots of smart fellers out there writing things about that may or may not be "important", but it seems to me that it is totally unimportant what the theoretical relationship of words in a database is, it's only important to understand the way searchers search and what they search for.

This is an outstanding example of why the KISS principle is so useful IMO.

[Steve Price]

After reading your post on Orion's post. I noticed that your attempt at explaining his formula is wrong.

If we are to take the n1 and n2 figures are correct for Dog, Canine and Dog, Pooch. There is no way that the n12 figure can be the same in both cases. I think if you got all your figures correct you would find that the Dog, Canine would produce a higher connectivity than Dog, Pooch.

[webcertain]

Hi there,
I believe that this article from Orion is not earth-shattering, but gives us a good opportunity to develop ourselves our method of keyword research. If you cannot be bothered (because you're # 18 for country music :-) ) fair enough, but selecting keywords is probably the most important task in an optimisation process so any idea to improve it, I will happily give it some times (instead of listenning to Joyce Seamone)

[Dave Hawley]

but selecting keywords is probably the most important task in an optimisation process so any idea to improve it

Agree, however, too many see this as simply picking the related term that is used most often. Gimmick yools like WordTracker etc take advantage of this gross over-simplification. Oh if only it was that easy!

[squirrel_4]

Agreed Dave, Word Tracker is somewhat inaccurate or over-simplified but, I feel has more relevance than the above and if not taken to the "T" can provide a good generalization into keyword selection(giving estimated searches and competitors). It should not be taken literally, because we know our users better than some automated program, but can help when torn between two phrases.

[Wrighthost]

If you look at the formula for jerrymac's site then you can conclude that his country music ranking is 0.0313 and country music singers ranking is 0.0019. This would mean that more connectivity would come from country music than country music singers, but it would be easier to rank with the lower connectivity figure than a higher? Just a thought!

[jamesljmv]

Using geometry arguments and fuzzy sets, it can be demonstrated that there exists an index, termed correlation index, c, such that

c = n12/(n1 + n2 - n12)"


I would like to see the "geometry arguments and fuzzy sets" discussion related to this formula. This looks bogus.

From basic electical theory for parallel resistances:

n12 might be considered similar to the product of two resistances, while n1 and n2 are the individual resistances. The formula is then:

c = n12 / (n1 + n2)

This would also be more sensitive to smaller "correlations".

[Peter (IMC)]

Jamesljmv,

With your formula the connectivity would never be higher than 50%, wo that wouldn't make sense.

If all pages containing K1 and K2 together are the only pages that contain K1 and K2 then it should be 100% since the K1 and K2 exist only together and not individual.

This would mean that n1 = n2 = n12. In your formula the result = 0.5

In Orion's formula the result would be 1.

There is no relation with parallel resistances. Orion's formula simply checks what percentage of the pages, containing k1 and/or k2, contain k1 and k2 together.

The theory that is behind it simply claims that the higher the percentage of 2 words existing together, the more likely it is that they are semantically connected.

Peter

[se-survivor]

I found the article very interesting, and a possible addition to keyword selection. I'm sure we will see a bunch of online tools pop up shortly based on this interesting information.

Unfortunately, getting the sales for a client is more closely related to monopolizing keywords that imply the visitor is ready to buy. Researching what those keywords are is far more important than "keyword semantic connectivity". Since I can already get first and second page placements, I'm far more interested in converting those positions into sales. If this means taking advantage of keywords that are not popular but convert to sales, then so be it. Those are the guys I'm usually interested in.

ex: putty vs "window repair"

"without marketing, SEO is useless"

[jamesljmv]

There is no relation with parallel resistances. Orion's formula simply checks what percentage of the pages, containing k1 and/or k2, contain k1 and k2 together.

The theory that is behind it simply claims that the higher the percentage of 2 words existing together, the more likely it is that they are semantically connected.


The parallel resistance calulation is actually derived geometrically, perhaps just as valid as the author's. The same form is seen in chemistry as well.

My suggestion was intended to challenge the application of the author's formula. Thank you.

[Peter (IMC)]

Yep, and I took up the challange,... :)

[jamesljmv]

Yep, and I took up the challange,... :)

Okay. What is the correlation to Orion's geometry and fuzzy sets? Would an RMS calculation be appropriate for three or more key words?

[Peter (IMC)]

Orion mentioned in his post:
According to Fuzzy Set Theory ("Modern Information Retrieval"; Baeza-Yates; Ribeiro-Neto, Addison, 1999), the degree of term co-ocurrence in a database is a measure of semantic connectivity (SM)
His formula calculates the degree of term co-ocurrence in a database, for 2 word phrases.

The correlation is direct. His formula just calculates that what the theory suggests. Whether or not that theory is correct is a different issue.

Your formula calculates something else.

For more than 2 words you still are after the same things I guess. Percentages,.. so for three words it would be something like this:

k123/(k1 + k2 + k3 - k12 - k13 - k23 - k123)

I didn't really put much thought in that so I could be wrong.

Cheers,

Peter

[reticomm]

Peter (IMC), I'm not so sure your formula will work.

One example: If you assume that the only time any of the keywords appear (k1, k2, or k3), are together and k123 = 100. This would mean that k1, k2, k3, k12, k13 and k23 would all also be 100.

C would equal -1.

[reticomm]

If the formula could be revised to account for the number of searches performed for the keywords, it could possibly have some value.

Perhaps a weight placed on the search terms based on the number of times the term has been searched (using tools such as word tracker and others).

That would make it interesting for me.

[Peter (IMC)]

Reticom,

You are right,....

It would need some more thought,. :D

Thanks,

Peter

[PositivePositioner]

If we are discussing how best to select keywords, it’s not all that difficult or scientific. Talk a few things into consideration and understand every website is unique and has individual goals.

Budget
Products, Services, or information offered
Target Audience
Supply vs. Demand

Budget: Budget is important because maintenance can be tedious and may have cost associated with the process. Also important to consider budgetary constraints regarding featured listings and PPC model marketing.

Products, Services or information offered: This is where you begin to construct your extensive list of potential keywords & phrases. First think of one key phrase that would sum it all up, that will be your site theme. For example: If you were a local Minnesota law firm your theme might be “Minnesota Attorney”. Don’t spread your selection to thin. Focus on your primary phrases until your site achieves positive positions, then add “landing pages” to include more key phrases. Try to limit one primary key phrase per landing page.

Target Audience: There is a big difference between visitors and your target audience. Visitors stray, while your target audience stays, pays and returns several more days. Who are they? Are they local, national, or international? What are they searching for? What is there age group? Define them best you can. Think like them. What would they like to see on your site, what do they want to accomplish?

Supply vs. Demand: This can be measured by checking a valid source such as Overtures “search term suggestion tool” (http://inventory.overture.com/d/searchinventory/suggestion/) to determine a rough estimate of how many times a particular word or phrase has been searched on within the last month. One at a time, type in your list of keywords and phrases and record the number of searches next to the phrase. You may also find many alternative phrases to consider. This is where a little experience helps. Because each client, website and situation are very unique. Next go to some of the most popular search engines and directories and search with those chosen phrases, and look at the number of sites that appear in the listings. Analyze some of the top ten sites. They are the competition.



Of course there is much more to obtaining positive rankings within the major search engines. Website content, functionality, popularity, ussability, conversion funnels, etc....

Feel free to ask me questions. Thanks.

[ronniethedodger]

I think some people are misunderstanding Orion's current posts at SEW to date as some kind of SEO formula, it is not. This is the first set of posts in a series that he will be writing about. It is the first step in more advanced relationships of occurance and semantics.

This first series is laying the groundwork for c-indices and nothing else. Everyone is assuming that it is some kind of "silver bullet" in coming up with the write keyword combinations (ala WordTracker). It is not.

If you have been following the thread you will see one inherit value with the k12 vs. k21 calculations very readily -- and that is what those calculations come out to be across different search engines (ie Yahoo, MSN, Google et.al.). They are all different.

Taking the "country music" vs. "music country" as an example of k12/k21 you might find that the k12 pair might be higher on Yahoo than it is at Google and completely the opposite for k21 (no I did not do the calculations ... I am just using it as an example).

Each engine parses content differently. The k12 pair can in fact be seperated by other words inside of the content and still qualify as being relevant for the search. Each engine has a limit to how many words may appear in between each k1, k2 pairings before it is no longer considered "connected". This is what Orion is demonstrating in this first series of posts.

He briefly mentioned placement of words within sentences, paragraphs, and distances. Take for example the following sentences (without any thought to relativity or semantics).

"Joe and I went for a Sunday drive in the country. Music filled the air from the tinny speakers of the old Philco radio in our beat up 57 Ford ... blah, blah, blah".

This indeed has nothing to do with country music ... it is music in the country or more basicly, the k21="music country". If one engine favors the k12 over the k21 in this case, then there is a flaw in it's interpretation of the content -- in other words it is evidence that they are taking words in a sequential manner without any regards to placement or connectivity.

People need to be debating Orion's formula (like jamesljmv did) and not whether it proves it will have any value in getting you better placement in the rankings. This one concept has nothing to do with better placement ... period. It is search engine basic indexing 101. It is laying the groundwork for the better stuff that is yet to come.

I might also point out that some people are following this for different agendas. Some are building (programming) their own search engines to work in conjunction with their Directories for instance. Plus the days of keyphrases in anchor text are quickly coming to an end and we need to understand some of this type of indexing right now (and not later).

[ronniethedodger]

Follow up to Garrets post. Orion is following this thread very closely. You might want to read his respononse.

http://forums.searchenginewatch.com/forum/showthread.php?p=1850#post1850

[Peter (IMC)]

Ronniethedodger,

Thank you for your post. This thread was indeed started from an SEO point of view. That the formula is not a magical SEO formula is obvious as well. I even mentioned it in a previous post:

It is my impression that these types of theories are of more use for ranking than they are for SEO. I would think that that phrases with a high semantic index will result in favoring pages that contain the complete phrase. That´s pretty much what we´re seeing already and what we would prefer in SEO.

I am curious about something though. You stated:

the days of keyphrases in anchor text are quickly coming to an end and we need to understand some of this type of indexing right now (and not later).
How is this semantic connectivity going to get rid of anchor text influences? I don't see how one compensates for the other.

Regards,

Peter

[ronniethedodger]

I am curious about something though. You stated:

the days of keyphrases in anchor text are quickly coming to an end and we need to understand some of this type of indexing right now (and not later).
How is this semantic connectivity going to get rid of anchor text influences? I don't see how one compensates for the other.


I probably should of phrased that a little differently. I was referring to the hundreds of 2-3 word text links repeated across many pages of a site ... kind of like 'keylink stuffing'. Especially on pages whose content has no relation to the page that the link is pointing at.

Identifying some links does not take a degree in rocket science either. There are ways of distinguishing navigation links from content links, that is pretty easy stuff. And so is that panel of "sponsor links" too. These are the kind of links that can easily be discounted if they appear judiciously throughout one site.

Of course this will vary from one engine to the next too. It is for certain that Google will diminish the importance of those links, while Yahoo may not give a rat's butt about it. MSN probably will go for the Google line of thinking (well more than probably they will).

The trick I think, will be getting those links in close proximity of the content ... and if inside of the meat of the page, they will be of most value -- regardless of what the anchor text is though, they will be links relative to the content. One only need look at a site's GoogleAds to get a grasp of what they think the page is all about. Follow a sponsor link from that page and if they have GoogleAds on it also -- do they match closely?

[Peter (IMC)]

Identifying some links does not take a degree in rocket science either. There are ways of distinguishing navigation links from content links, that is pretty easy stuff. And so is that panel of "sponsor links" too. These are the kind of links that can easily be discounted if they appear judiciously throughout one site.
I agree that certain parts of a page that are repeated through the whole site should be looked upon differently, though not be ignored or discounted.


One only need look at a site's GoogleAds to get a grasp of what they think the page is all about.Yep, I noticed that too, and when optimizing a page, those ads all the sudden match much better, and/or you get more ads and less alternative ads.


Follow a sponsor link from that page and if they have GoogleAds on it also -- do they match closely?
Should they?

Peter

[ronniethedodger]

Follow a sponsor link from that page and if they have GoogleAds on it also -- do they match closely?
Should they?


If they do, then no problem. The question, more properly should be -- should they not?

[Peter (IMC)]

If you look at a page with the idea that it can contain only X for content, then ads should be the same.

But I think that the content of a page is more like XYZ and an ad can be related to X, Y or Z.

Have you ever done adwords campaigns? If you do you will see that you are allowed to target wide ranges of keywords. They don't have to be exact matches with the page the visitor will be brought to.

Adwords works because of contextually matching ads and content. Trying to target phrases that don't match your ad will result in a very low click through rate. (proof that contextual advertising works). If the click through rate gets too low (below 0.5%) google will stop targeting your ad for that keyword phrase.

So how closely page A matches the ad shown on page B that you reached through an ad is not telling you much.

Regards,

Peter

[ronniethedodger]

So how closely page A matches the ad shown on page B that you reached through an ad is not telling you much.


I was referring to the clicking on the sponsor links, not the googleads. I can understand your confusion now. If the site with the sponsor link and the site it is pointing at both have googleads running, then you will have a pretty good idea of relativity would you not? That is all that I was saying, the googleads would give you a pretty damn good idea of how google perceives the two.

Heck I published some ads on a site that dealt with internet and web development information. All the pages the ads were on were fine, except one -- the bugger kept spitting out weight loss pills and how you can lose 18-27 lbs in 6 weeks !!! I took that page down ... still could not figure that one out for the life of me.

[Garrett]

I seem to have made some grave errors in my calculations and oversimplified/misunderstood the formula as it was originally presented by Orion, as he points out here (http://forums.searchenginewatch.com/forum/showpost.php?p=1850&postcount=83).

So I'm hitting the books again to try and understand the basics.

Thanks to Orion for visiting and correcting.

[AussieWebmaster]

One of the better articles/discussions in the industry at the moment... it is good to see many diverse people keeping it going and making others aware of it.
Garrett: I had to read through it 3 times before I started to have a clue... and am still finding maybe that was an illusion... but it's good to have something else to discuss other than importance of title tags etc.

[ronniethedodger]

I am with you AW -- it is a very nice change.

I can't wait for the book "Keywords, Co-occurrence & Semantic Connectivity For Dummies" to come out now! It will definitely replace my "PHP for Dummies" that is presently sitting underneath my cup of coffee all the time. ;0)