New Search Engine... Who wants to help design the algorithm

[collusion]

I am in the process of starting a new search engine. When you go about a task like this you have to way many options when writing the algorithm. I have created a system to go by and unfortunately it is classified so it will be of no help to you to figure this out. I would like to know everyones take on how their ideal SE would work. Basically, what really ticks you off about Yahoo, Google, MSN, AOL and any other biggie. My reason for this is to get ideas while I am creating this algorithm. I want to be fair. I do not think a "sandbox" is really fair unless it has a set of boundries that will allow webmasters to get high rankings as long as they are not trying to spam and have relevant content. This algorithm will be based on "THOUSANDS" of items which I cannot go into. Please tell ALL.

[Maloney]

Things that tick me off about Google:

Link spam works
CSS cloaking works
Cloaking works
Results are stale - new sites are ?? where exactly

Good luck with your new engine.

:)

[collusion]

Thank you for your response. I agree that cloaking works with Google. I also hate that companies can but IBL's and Google depends on them, even at the slightest bit. CSS huh.. I do believe I know of a few sites that rank well using CSS Cloaking. All of the pages that are ranked have a php extension though.

Also, the sandbox just don't work. There are companies that started yesterday that I would trust more than companies that have been around for years and unfortunately, the new company is nowhere to be found until months after it was indexed. Horrible stuff.

[webmasterjunkie]

Good luck with the task you are setting out to do. I thought I would give you my 2 cents worth as well.

MSN :

IMO, MSN seems to have a problem with subdomains. If a site is highly optomized then when you search for something, the relevant page is in No. 1 - where it should be, but the next 3-4 results are for non-relevant pages on subs for the same site.

Google :

Of course the sandbox is ridiculous. As you said, with no good parameters set, they hold back on the sites that people should be seeing.

Yahoo! :

The only thing I can say about Yahoo!, kind of applies to Google as well. The results have a tendancy to run stale. Like the bot doesn't go on to many field trips.

Hope this helps.

[SEOforGoogle]

[quote]I do believe I know of a few sites that rank well using CSS Cloaking. All of the pages that are ranked have a php extension though.[quote]

You can also css cloak without the php extension, you should wipe that out, and also, I still have no idea why the SE's love the h1 tags - maybe give higher rankings to sites that have the page title relevant to the actual content on the page?

Good luck and keep us posted.

[timmathews.com]

The main factor that bothers me is the amount of time Google.com, Yahoo.com and MSN.com take to actually get you indexed.
If you can write your se to do immediate indexing, inclusion and add to the search results, I think this would make you stand out.

This way, the "old dogs" that we see everyday when searching that are not relevant will not be given more weight than a new site with better content.

I would be really interested in beta testing your se.
How far into development are you?
Can we have a sneak peek?
Thanks!

[joseweb]

I am not much help in the algorithm department, but I do spell "weigh" instead of "way" when I am talking about a decision. I hope you are better at writing code than spelling.

:)

[collusion]

I will be happy to announce any beta testing to those who email me about it at brandon@ageofdesign.com

I cannot give any news or specifics yet but I can guarantee it will compete heavily with Google.

[collusion]

Originally Posted by joseweb

I am not much help in the algorithm department, but I do spell "weigh" instead of "way" when I am talking about a decision. I hope you are better at writing code than spelling.

:)

LOL, didn't catch that one. Thanks for the correction. You will be glad to know that I have programmers on top of it for me. It is hard because I cannot kill off anyone who knows the secret algorithm so I have to have more than one programmer working on different parts of it.

[arpecop]

1. buy 800 HDDx 80 gb and index only 2k from every page (only meta)
2. congrat! you have mini google :)

[collusion]

LOL, Funny. My budget is a bit bigger than that though.

[Webpub]

I do not mean to cause any issues here but if I were Google Engineers, I would come on here ask/state the same Questions such as "I am starting a new search engine" so that I can find out what ya all are doing and then counter it.

So you tell the big search engines how you are optimizing and speed the process up of loosing your own job optimizing or gaining any momentum from new hits.

Why does everyone want to tell everyone everything?

We have pretty good positions on the major SEO terms online, we would never tell someone how we do it, why, it is our profession and we guard it. Why would we want to tell Google in a public forum how we optimize?! or someone we don't know named John Doe?

[Webpub]

Collusion - what is your budget? We are all sharing with you, share with us.

[collusion]

LOL, you are funny too. I am no Google employee or affiliate with them in any way. I actually hate Google. My budget is confidential. I am sure if you were in my position you would understand the need for confidentiality also.

About you giving away secrets. I have personally ranked high in EVERY major search engine for my clients sites. You have no "tricks" that I do not know about. The reason I ask the above is because I want to know what draws people AWAY from the search engines. What I can do to make it a place you would want to make your homepage. I am not necessarily going to do it just because you posted in the thread. I am going to weigh what I know about SEO, and what I know about shady SEO along with what keeps visitors to my search engine. I am looking to bring relevance where no other search engine has been able to. Don't say Google has either because there are a number of people here that can show you spam sites in the top 10. If you have any useful info please share.

[Webpub]

I do not opt-in to share as stated for my reasons above. (IMHO)

Good SEO is SEO that is not made public, I do know things you do not. (IMHO)

If there are no tricks you do not know about then you should go start your search enigne, why bother asking people here.... seems you have most the info you need.

;-)

[collusion]

AGAIN, I am looking to find out what would make you stay. Please do not post if you are not providing anything related to the topic.

[collusion]

p.s. You can continue to think I do not know what you know. I will keep you in mind when I beta test to find out how easy you can rank to the top on my search engine.

[timmathews.com]

Collusion & Webpub
Chill pickles.
This is not a pissing contest.
Lets keep thread replies relevant to the topic please.
What ticks us off about the big dogs.
Who cares if he works for one of the big dogs. IMHO, I doubt he does for his total lack of PR for his 3 sites in his signature.
;)
Sorry about the bash, but I am making a point.
If he does, COOL! Maybe we can actually voice an opinoin that will mean something.
Cheers!
Tim Mathews

[timmathews.com]

Originally Posted by collusion

I will be happy to announce any beta testing to those who email me about it at brandon@ageofdesign.com

I cannot give any news or specifics yet but I can guarantee it will compete heavily with Google.

Thanks bro.
I have emailed you.
Looking forward to testing it.

[collusion]

Not mad atcha at all. Those sites that you mention are reletivily new and I have not been promoting them. EXCEPT for AOD which has a decent PR in the amount of time I have been working on it. I want to keep this to the topic at hand. Please dont go off topic. I am really interested in viewpoints people may have about why they wouldnt use a specific search engine unless they HAD to for work. What would in your mind determine a websites relevance. My algorithm is determined by over a thousand factors. I go into great detail in the algo so that I am SURE that the site belongs on the top without manipulation. Matter of fact, some of my personal websites if not all will most likely never see a top 10 position in my search engine. I better buy myself some adspace :)

[vitamins.com.au]

collusion, why do you ask SEOs what they want from a SE ? don't you think that SE's are made for users ?
If you ask SEOs, they are likely to tell you what they expect a SE to be like so they can rank well and you can't blame them for being biased (I certainly am).
This makes me think that your post is not serious, otherwise you would make your market analysis with the right demographics.

[collusion]

I am a SEO myself. I was hoping to start a serious discussion here to get some ideas. I know how to filter out spam and figure out how sites are relevant or not. I want to be sure that I have it polished off so it is better than any other SE you may come across. If you do not want to share, don't. I will still develop it anyways. Just thought some people may be interested in taking part in what might become a great history.I understand some posts may not be toward helping making the SE better, but influencing the results. I am well aware of how to figure out what is relevant and what's not. Post away, your suggestion may somehow be included and it may not. We shall see.

[Maloney]

Two thoughts about what I LIKE about Goog:

It's clean, lots of white space, not cluttered, plus 99% of the results page is filled with - results! (that part applies to yahoo and msn as well). I do prefer goog's paid placement off to the side though, not like msn and yahoo with their ads on top.

Fast, the results load lickety split

What I don't like about Yahoo and MSN:

paid ads are first (even as a normal user it feels like a pesky fly I want to swat away)

[Webpub]

Unless you have a 51 Billion budget (Current Google Valuation) then I am not worried at all.

Money is what you need, you need Billions to compete now, can you come up with even 1 billion?

I am not trying to be fresh but Bill Gates & Sergey have plenty of cash to discover who is up and coming and make them a small-fry.

There are barriers to entry now, it was not like this in the market back in 1997, it be 2005 dog.

Let me play Devils Advocate for a bit.. it is healthy.

;0) -WebPub

[collusion]

Originally Posted by Webpub

can you come up with even 1 billion?

You will be surprised. Still, you are off topic though. The topic isnt whether I can afford to compete. It is about the algorithm giving relevant results. I can care less what resources Google has that you THINK I don't have. If you have relevant results and they are better than Google, word will spread and people will follow. Please stick to topic. Thanks.

[dmgreer]

Something I hate about current search engines is news item flooding, or whatever you want to call it. Suppose I'm looking for US Civil War information, and I look up the battle of Appomattox and everything is fine, but then a movie called The Battle of Appomattox comes out. Now the first 10 pages or more are about the movie.

Somewhat related is content flooding. Suppose I want to find out about knee injuries. So I look it up and I'm flooded with basically three or four pages that have been copied over and over, and maybe there are five other unique pages buried among the rest. I just want to see unique content and avoid the repetition, or have the repetition organized at least.

Am I making sense here? To me the great disappointment of the web is that it is an excellent tool for disseminating bad information. Someone can put something out there with little or no research, and if for some reason it becomes popular, then everybody will copy it, and before you know it, good information and differing viewpoints are buried in a sea of copies.

I don't know if there's anything to be done about that, it seems like it would be computationally very intensive.

[collusion]

Very hard to filter out but doable and you just made my priority list.

[AcmeSEO]

Back in 2000, I wrote an algorithm for my pet search engine, Alaska's Search Engine, AlaskaBot. Yep, you guessed it, www.alaskabot.com. It works pretty darn good. If I ever get time I'll refine the algo, it needs some refinement. Right now single word searches yeild 98% relevant results. If you are going to Alaska use it for info, the database is impressive. I have big ideas for it yet, but right now, SEO occupies my time. And I love it! I am trying to optimize each aspect of my life, in great part due to SEO.

A new Search Engine? Right on brother!

Peace,

Robert

[AcmeSEO]

And... what do I think would make a good search engine?

Well, in a perfect search engine, white hat works and black hat don't. Casinos, porn, and drugs would not appear in organic search, but in paid search and subscription only. And paid advertising (PPI and PPC) would be based on principles of revenue and profit sharing.

I would dig that search engine!

Peace,

Robert

[espectations]

What I hate:

1) Spamming sites that are ranking higher in
results than decently optimized sites.
2)

[espectations]

What I hate:

1) non-relevant search results.
2) Not all of us can afford pay-per-click
advertising and if you cannot afford it you just
do not rank high enough. There should be an
option for smaller companies that do not have
billion dollar ad budgets. The world does not
just consist out of America - with the exchange
rates we cannot afford to pay for advertising
because your dollar is 6 times our currency.
3) Link farms that seemingly rank higher than well
optimized sites.
4) Frequency of crawls not frequent enough - I
believe G crawls once a month so what about us
that update our sites daily? (correct me if I am
wrong pls)
5) Cached results which you struggle to get rid of
6) Most engines I see is now becoming regional. It
works on the IP aparently. Not sure if that is
such a good idea because we need to market to our
target markets which are not always regional.

I will mail you too and would like to test the new engine - competition is always good.

[simonm]

Human intervention might be needed in some areas. What I mean is, say I do a search for 'police london' its clear I want a result with london police websites. If the 1st page is filled with 'The Police' and their recent tour of London, well, frustration sets in.

I guess you might want to automate this so perhaps a word association algorithm as part of the system. I do a search on police, the top web page (site) results have words that have a relation to the search term. eg pages with constable, sergeant, crime, victim etc. would be prioritised.

I expect a decent bit of AI could generate the word association lists automatically from the sampled (spidered) web pages. Then the word association lists could be totally neutral and just show patterns rather than direct associations with subject. ie Y word normally has Y1 Y2 Y3 Y4 words with it, Y1 being the most likley word to be associated with Y. Thus a page with words Y, Y1 and Y4 is more likely to be a valid result than a page with word Y (a word in the search term) and no other associated words.

I hope the above makes sense.

[smo]

If I have some content to add on an area and I do so by creating a new site or to a site not so high PR ( discount sandbox here ) then my chance of getting good listing is less.
But if I am adding the same content or poor content on that area to an existing good PR site then I will do well.

So the importance of site is high than the importance or relevance of content. Why it should be like this?

[TrafficProducer]

What really ticks you off?

Not being able to find what I want, searching for.

Too, too, many adverts getting in the way. (But then how will you make money).

How about this idea:

1) I search say for 'Green doors made in plastic with no glass'. Nothing found. Your search returns suggestions for other words to search on e.g.
'Plastic doors in the colour green with out glass'

Note; just a text suggestion not a load of irrelevant sites.

2) Willing to list my site a number one on all pages without charge. Some hope!

Good luck

[MathsIsFun]

A learning search engine.

Using the "london police" example above, the search engine could build up associations to the point where it could ask you if you wanted police (that protect) or police (that make music).

The associations could be learned from the users, picked up from tags in web documents, and could be jump-started by spidering via DMOZ.

[greeneagle]

Simonm:

"Human intervention might be needed in some areas..."

Very astute observation Simon. A recent WMW thread estimated that engaging existing entities in the SE World would require an estimated $15M (USD) in capital expenditure alone to become competitive in only one subject; "Sports" - all by itself!

collusion,

Since you don't see "eye-to-eye" with DMOZ... Which leading open resource (free) directory do you intend to partner with for initial resources?

Of course you have made indications that monetary resources may not be an issue... so where is your original "Hot POP" DB coming from? Surely you have already made those arrangements, this far down the road!

There are no secret "open resources" are there? That is the only logical place to start, right?

If you have a vision that bypasses the existing, which indeed you might... just toss a bone here so that readers will follow you to your new SE!

I would suppose that you wouldn't have opened this thread without it being an initial marketing endeavor... Am I wrong?

If you indeed have a better plan than GOOGLE, I'll follow your lead, and support you whole-heartedly!

Best wishes - But IMO - we need more "Pied-Piping" to go along here!

Ken

[collusion]

Ken, what he said about starting with DMOZ isn't a bad idea. Whether I like them or not, or even agree with some of the indexed sites, what does it matter if I start with the links and filter them out with my algorithm? The point is not where I am going to start with content. Content and URL's are very easy to collect. Any spam sites that I get from another source will easily be filtered out when the spider hits the road runnin'.

[wkennedy]

Some features I feel are needed or should be improved:
1) real drill-down capability (search within results feature has disappeared from most se;)
2) "show similar pages" should actually do what it says - amazing to me that if i enter "chicago apartments", click for similar on a condo/apartment site, i get the city of chicago! Filters here should be smart enough to know better.

features to be retained and/or expanded:
1) search hints (Yahoo! does this best;)
2) filter options in prefs - i.e. Alta Vista's family, regional & language filters

biggest turn-off:
Enter the word "free" in any search and 99% of returns try to sell you something. What's up with this?

[greeneagle]

collusion,

I am just not sure what you have said besides you intend to "crank up a spyder" here. Any one of us can do that tomorrow!

Could you be more specific than that? There is a gaping chasm to fill in with suppositions, without violating propietary issues there!

Ken

[collusion]

GO do it then greeneagle. DO NOT turn this thread into one of your DMOZ threads. Keep to the topic is all I ask. What do you have to provide to make the quality of search results untouchable?

[collusion]

wKennedy, I agree with all of your issues. The SE definately needs to be smart.

[greeneagle]

I didn't mention DMOZ, collusion, all I was asking about was primary precipices. What is your primary approach? How are you going to get past the awsome competition? What is your angle? What is your secret approach that can possibly imagine bypassing millions or billions in research? How were you going to start? Who and were you going to draw from (which Directory Source) to get a start?

That's all!
Ken

[collusion]

Ken, the inner workings of this SE is classified until I can announce beta testing. This is the policy that has been set and I intend to keep it there. Sorry.

[collusion]

P.S. - Feel free to email me at brandon@ageofdesign.com if you want to be a beta tester. I already have a ton of emails for it and I started it's own folder in outlook express.

[jestep]

If you're going to make something great, I suggest to start with Latent Semantic Indexing. Work backwards from there until you get the results that you are looking for. You aren't going to have a breakthrough using the same thing everyone else is using. Google, MSN, and Yahoo have invested millions of dollars in their search technologies. Recreating the same thing isn't going to get you anywhere.

After reading several documents about LSI, I am convinced of 3 things.

First off, most search engines are going to be using LSI in the future.

Second, is that it is the best way to prevent spam in any search engine, while keeping the most relevant sites at the top.

Third, you probably need a doctors degree in AI, and technological systems to even remotely grasp how to program such a search engine. Maybe a team of doctors would be better....

Here's a great post on LSI
http://forums.searchenginewatch.com/...ghlight=Latent

[greeneagle]

collusion:

"Ken, the inner workings of this SE is classified until I can announce beta testing."

What is your projected date for "BETA", surely you have had to inform your investors and maybe even the public concerning that issue?

I am looking forward to testing it for you. In fact I'd like to get in on the "ALPHA" testing, right now, today!

How can I do that? Anything better - I gotta get there!

Ken

[collusion]

I have a few dates set up but in the worst case scenerio the beta testing would launch in December of this year.

[greeneagle]

What happened to "Alpha"? It is a tradisionary custom to release an "Alpha" before "Beta". Is "Alpha" already engaged and locked-in?

If so, how's it going?

Could you provide an "Alpha" link like all the other SE's so someone, anyone can test it?

Ken

[collusion]

I am not giving public alpha testing. That is being done by internal staff.

Alpha has not yet started either by the way.

[collusion]

greeneagle, are you going to keep grilling me or are you going to provide some useful information like alot of others have already?