Search Will Die From Chronic Obecity

[Ferluci]

A few years ago a "culture" became established in the world of search that I believe will ultimately destroy the major search engines and possibly the major directories. The concept that "content is king" was a nice idea, but what we actually got was "VOLUME of content is king" though I would argue even that... Then the notion was introduced that inbound links to a site effectively constituted "Votes" for the sites suitability in relation to basic elements used in the text of those inbound links - so links became king.

Upon realising what was going on, SEO pro's started to try to take advantage of this new trend in search and started to use various means to increase their sites content volume and (probably more crucially) the volume of links (votes) pointing to it. Some resorted to duplicating content in blocks and found themselves penalised for that (sometimes unfairly) and a mechanism was introduced to "hold-back" new sites that appeared to gain new inbound links too many and too quickly (sandboxing).

Since then all the major Search Engines and some of the directories have introduced various extended measures to avoid this bloating effect (to be fair - the directories didn't start this war). The major search engines, in particular, have had to make major changes to their infrastructure to cope with the increased levels of demand that they themselves created.

It didn't have to be that way... If instead the position was taken that only the first relevant (say) 100 Kb of a site would be spidered and only the first (say) 100 inbound links found relative would be taken into account - this combined with other relatively "do-able" limiting factors would have reduced the overall size of the indicices in general and reduced the amount of information a searcher would have to wade-through to get where they wanted to be...

I was very interested to read of the directory attempt That Webnauts and colleagues are making in this thread Webnauts Directory Project I think some of such criteria could by checked by bots though it is a little involved and could not be done to the same extent using AI.

In short - I think we could have a much more concise and usable internet if the big players hadn't got fixated on volume...

What do you say?


Kind regards

Mike

[kgun]

Originally Posted by Ferluci

A few years ago a "culture" became established in the world of search that I believe will ultimately destroy the major search engines and possibly the major directories. The concept that "content is king" was a nice idea, but what we actually got was "VOLUME of content is king" though I would argue even that... Then the notion was introduced that inbound links to a site effectively constituted "Votes" for the sites suitability in relation to basic elements used in the text of those inbound links - so links became king.

Upon realising what was going on, SEO pro's started to try to take advantage of this new trend in search and started to use various means to increase their sites content volume and (probably more crucially) the volume of links (votes) pointing to it. Some resorted to duplicating content in blocks and found themselves penalised for that (sometimes unfairly) and a mechanism was introduced to "hold-back" new sites that appeared to gain new inbound links too many and too quickly (sandboxing).

Since then all the major Search Engines and some of the directories have introduced various extended measures to avoid this bloating effect (to be fair - the directories didn't start this war). The major search engines, in particular, have had to make major changes to their infrastructure to cope with the increased levels of demand that they themselves created.

It didn't have to be that way... If instead the position was taken that only the first relevant (say) 100 Kb of a site would be spidered and only the first (say) 100 inbound links found relative would be taken into account - this combined with other relatively "do-able" limiting factors would have reduced the overall size of the indicices in general and reduced the amount of information a searcher would have to wade-through to get where they wanted to be...

I was very interested to read of the directory attempt That Webnauts and colleagues are making in this thread Webnauts Directory Project I think some of such criteria could by checked by bots though it is a little involved and could not be done to the same extent using AI.

In short - I think we could have a much more concise and usable internet if the big players hadn't got fixated on volume...

What do you say?


Kind regards

Mike

I have the following comments:

1. To me content is Queen and refactoring to better content is King. But good URL's never die. I can not see why content is not King. It is said, that Newton produced about 50 % of the contributions to mathematics up to his time. It would have been bad to lock out "todays Newton", because he could not write valid HTML. If you want to send a spaceship to Mars, that spaceship may land on the moon if you use wrong information. Code can never be more important than content in my view and for my use. As far as I have understood Webnauts, he has to make the strict rules because of disabled persons, pages shall be viewable in text based browser like Lynx, and on mobile media. There is in my view a clear niche for such a directory.

2. The SE's can link millions of pages before human beings have discovered them. Some SE's give more reliable results than others and still others prefer a Mazda to a Lamborghini Diablo especially older people. There are special cars for disabled people too.

3. The best SE's will never die, but a lot of directories will in my view. The digital revolution has just started, there will be a new internet2 that is 100 times or more faster then todays net. I saw a speech on CNN, where Bill Clinton, when he was president, talked about the next internet that would be 1000 times faster then todays. Pages that today loads in 15 seconds, may then load in fractions of a second.

4. Human beings have advantages to BOTS, eg. pattern reckognition. But human beings can never compete with the speed of a Bot. Directories are more complementary than competitive to SE's. They serve different purposes.

"For the most part, human-edited directories like DMOZ and spidered search engines like Google serve entirely different purposes: Directories index sites, and search engines index pages. It's like the difference between a library's directory of the magazines in its collection and The Magazine Index or the Reader's Guide to Periodical Literature.

If you're looking for a site about Elbonian travel, you might use a directory. But if you're looking for an article about mud kayaking in Elbonia, you need a search engine like Google. That's why human-edited directories can't be a substitute for spidered search".

Posted on the webmasterworld forum may 30, 2005 by the owner of europeforvisitors.com.

5. May be the SE's that get the best spam, scam, phising, pharming filters etc. will win. Some cars are more safe than others. I do not know how many use Google advanced (image) search with the SafeSearch filter on.

6. In Norway, there are two directories (portals)
- Varslingslisten (Warning list) and
- Trygg katalog. (Safe Portal)

The last is definitely against the free spirt of the interent and for the guys with big pocketbooks. Is it possible to imagine (a lot of) mafia sites on a similar global portal?

Do you think the facists in Spain liked Salvador Dalis art?

I know, I sit in glass house myself, since I use strict criterias for inclusion in my niche linkcollection.

There is nothing freer than free searh on a good Search Engine. My working horse is still Google until another horse has proved better. I doubt that will be a portal, that will be more like a regatta without wind.

In the blind's world, a oneeyed person is King :-)

[Ferluci]

My arguement was really about the unhelpfulness of bloat in the world of search. The fact that a new faster internet is under development does not IMO make it ok to have massively expanding indeces - they will just get fatter faster...

I am concerned for the little guy who has a small company, with a small range of products, who can't really find a practical way to produce a site with 2000 pages about "wingnuts" and is even less likely to find 200 "wingnut-related" sites to link with.

These days directories are something that we use to get good IBL'S pointing at our sites, most people go into a directory listing via a search engine listing that pointed them there. The basic principles behind a well tested and researched directory listing could be reproduced in search engine listings if the indices weren't so fat - so the secondary bots could spend more time doing things like code validation.

I still think the opportunity is there for a group to develop something small and beautiful that could develop a following amongst searchers that goes well beyond niches and becomes the homepage of many such searchers.


My eyesight is fine by the way...

Mike

[kgun]

Agree to that. My cite about the blind was general and not aimed at you :-)

Stephen W. Hawking's web pages is one of my favourites:

1. Minimalistic front page.
2. Text-only version and a graphic version.
3. If you have any problems with accessing this site, please contact ...

If I had the necessary resources, I would have done that myself.

[Ferluci]

Fair enough...

Don't get me started on accessability and legal contact requirements in EU law - that' a whole other thread lol...

Mike

[kgun]

Some Swedes (member of EU) call Norway the last Soviet state. It is more important for Norway to beat Sweden in soccer than win the world cup.

Switzerland is more independent from the EU than Norway. When we study the footnotes from EU law and regulations in greater detail, I wished Norway preferred the Swiss solution.

But there are some good rules from EU, especialy EU control regarding the safety of cars.

[Ferluci]

There are some interesting EU rules regarding websites - most of which get ignored....

(links reserved for another thread)

Mike

[freetraff]

Originally Posted by kgun

Stephen W. Hawking's web pages is one of my favourites

Thanks for bringing up Mr Haking's web site. I admire him, his achievments and love to life.

Plus web site is great. That's a good example of creating just a site under reasonable laws and it naturally becoming very nice even in terms of SEO.

[incrediblehelp]

I would rather find a bloated website full of relevant content, than a directory website within the SERPs. I felt for a long time that it is a not a good experience for a searcher to search a search engine and find another search result page, as they do when finding directory pages. Now-a-day's a lot of those catch all crap directories are no where to be found and it is a good thing. Of course the more niche directories, somewhat like Webnauts directory, should be found if their are voids in the current database of index pages by the search engines. Seriously how often does that happen? I am sure a web page has been written on every subject out there and more content is added to the website everyday. This is a good thing. The search engines are not bloated, they are simply not good enough at what they do. It is just the search singes fault that they could possibly not be able to index all of of the web and then rate it properly. Limiting indexing by crawling only the first 100kb of a website and 100 links of a website can not true and does not make sense as a solution. If so please prove it to be so. I am would very interested to see how the engines are doing this and also why would they do it.

[kevan]

Just an amusing quickfix for Bill:

Hi,
I run a genealogical Pub site for London at http://pubsinlondon.net - it has about 5,500 individual Pubs, plus lots of listing pages. Google appears to have it, as does yahoo.

Good luck with your search engine, I am sure it just needs some tweaking.
Best
kevan Wilding

posted to the MSN search engine comments site.

Best
Kevan

ps I love WPW, it is a brilliant site

[Duncan Pollock]

There's a point that perhaps we're all missing and that's what the purpose of the Web -- as opposed to the search engines -- is and I believe orginally was: to enable everyone and his brother, sister, nieces, nephews, and whoever else to create a site that people could refer to when, if, and as they choose. All that was required was the technology that made the creation and uploading of sites possible and this, among other things, came down simply to the allocation of an URL, which could-be/would-be visitors could be informed of by one means or another.
But then enter the search engines with an avowed purpose of indexing as many sites as possible and then organizing them in such a way that people who knew absolutely nothing about them could reach those that represented (by search engine standards/opinions/algos/filters/whatever) the most appropriate/relevant response to a given/imagined search keyword or phrase.
The two purposes are quite different. Their origins aren't the same and the degree (or not) of satisfaction is going to vary accordingly.
On the one hand, assuming that I've been able to create a site and told someone what its URL happens to be, the web should find and display the site -- and, from all my experience to date, this happens.
On the other hand, if I want people to find the site when they aren't aware of its URL (or don't even know that the site exists), it's up to the search engines to come forth if they can or are disposed to do so -- and herein lies all the mutterings about SERPs and the seeming insensitivity of the guys/gals who program algos that end up with spam, irrrelevant sites, and -- so help us -- NOT the result (i.e. our own site) we want to see in the Top Ten.
So I don't think it matters one iota whether the web is bloated or not. If anyone and his brother etc., etc, has the ability (or at least the opportunity) to have and upload a site to the web, let's be truly thankful for it and acknowledge that the results are everything they were intended to be from Day One.
And if the search engines frustrate us with their apparent unwillingness to behave in our site's favour, then such are the breaks and we surely have to accept this fact regardless of how unpleasant it can sometimes (too often?) be.
We can't be all things to all people and we can't expect Google, Yahoo, MSN, et al to do our bidding simply because we want them to do so. It's enough that our sites DO exist. The rest is up to us -- and, thanks to WPW among others, help in this regard IS at hand.
(End of rant and pontification)

Duncan

[wwwizzard]

Well, this is what I think ...

Once upon a time people built websites for people.
Then along came the search engines and people started building for search engines.
So if you want to find a website that is built for people, put in your keywords, and work from the back of the results for the best sites!

(Only half joking)

John

[Rosalind]

Bloat is a problem. There are billions of pages out there, and bloat is what gets in the way of the specific information we need, and makes everything harder to search for.

Most of us are guilty of contributing to the problem, to an extent. Every time we make small talk on a forum, or waffle in our articles, that's bloat. Every time we intentionally fluff up our pages with extra keywords, or inflate our page counts with superfluous pages that aren't really necessary. The worst thing is whenever anyone builds a huge scraper site, or other random-text spam.

Behind all this are the search engines, and this whole proliferation of pages is the direct result of their algorithms. If they were only capable of recognising and rewarding concise writing and site structure, there would be no bloat. But instead they are hugely biased in favour of large, unwieldy sites with lots of inbound links.

If you could build a bot capable of assessing the ratio of real information to waffle, it would be worth a fortune and the problem would go away almost overnight.

[incrediblehelp]

Bloat is not the issue, it is trimming the bloat or deciphering it that is the issue. This is the responsibility of the search engines.

We as web designers and website owners should stop building content? Not.

[chrisJumbo]

A big problem is most people want something for "nothing" or don't have the funds to muster a true advertising campaign.

You can't expect the SEs to spend a lot of time programming to give you the results you want without making any money.

The SEs need to recoup their costs and they each come up with a business plan to accomplish this.

As far as the SEs limiting how much they search of any given site and the #of links they count, what happens when everyone figures that out and designs their websites to follow. Who gets to be #1 when everyone is the same?

It is funny, that when we rank where we want to rank for our keywords we are all happy. But when they drop us they are bloated and spammy. And I live in that realm as well so not pointing fingers.

As long as people want to find the "best web designer" or "best widget maker", SEs will exist. And web designers and content masters will have to play their game, unless we want to spend our time and $ to make a better one.

[murphypj]

Once again, I find myself in total agreement with IncredibleHelp on this issue:

I would rather find a bloated website full of relevant content, than a directory website within the SERPs.

I can also identify with Duncan Pollock's points - I'm guessing we both uploaded our first fledgeling webpages at about the same time - when the web was about creating content that would ( hopefully ) be deemed useful by those who visited, and before it became the seedy side of commerce, and 'black hat' was something the wife wore to a wedding...

I was lucky enough to get an early high ranking with Google - happy accident back then, but thanks to developing SEO savvy thru newsletters and fora such as WPW, the continued top placements are now down to hard work - plenty of fresh content and keeping up with what the SE's 'soup of the day' is at any time, and I received my 10 millionth visitor earlier this month.

It is sad enough that I devote more time these days to keeping up with and deploying good and clean optimization techniques than I do on creating new content, without being made to feel guilty because my code does not meet with certain standards, which might be said to be the luxury of those with the budget to uphold them.

I make every effort to minimalise my code - for page load time reasons - but have no control over the affiliate code ( including Google Adsense ) which I need to make the site viable.

As previous posters have said, good and plentiful content should be king, and it is up to the Search Engines to provide searchers with the best results for their enquiries, bloat or no bloat.

It was the provision of regular relevant results which elevated Google to it's current position - both in financial and public perception terms. Google needs to deal and dispense with 'fat code' issues itself, and continue to satisfy it's customers, to maintain it's position and marketability.

I have every confidence that it will.

PJ

[kgun]

Originally Posted by incrediblehelp

This is a good thing. The search engines are not bloated, they are simply not good enough at what they do. It is just the search singes fault that they could possibly not be able to index all of of the web and then rate it properly. Limiting indexing by crawling only the first 100kb of a website and 100 links of a website can not true and does not make sense as a solution. If so please prove it to be so. I am would very interested to see how the engines are doing this and also why would they do it.

And never will humans be able to rate the books in the world. I am sure that my top book is different from yours. And even if we agree on the book, we will rate different parts (pages) different.