Nutch - on popular demand

[Conficio]

Hi everybody,
I have mentioned and advocated for an open source search engine called Nutch in these forums before.

The current status of this technology development project has not reached a public demo. That has brought me some critic, as if it would be less real and or useful.

I thought I should report, that this has changed (somewhat). Although the project still does not announce any public demo, some of their backers use it in their research labs and allow public access.

It does not amount to much, as the index is really limited. But it does proof that the technology has its use and purpose. The argument I'd like to repeat here is, that this will be influential for search engines, because it allows anybody with an idea to implement it at little cost. This should foster more alternative of search engines and may be more specialized engines.

Happy surfing
K<o>

[minstrel]

Interesting...

The search results are definitely quirky at the moment but it bears watching... especially if Yahoo and Overture are taking it seriously...

[davidshq]

If I understand correctly, Nutch would allow us to basically understand the ranking algorithm because the ranking and source code to the engine itself would be open source? What about the database? I like the idea of an open ranking formula - but think that it can't be used solely to support an engine as individuals with knowledge of the ranking formula will easily be able to drive top rankings of garbage sites.
Respectfully,
David.

[Conficio]

High David,
I'd like to point out a few things.

The licenses of Nutch allow the use and incorporation of proprietary ranking algorithms. But what it supplies is an efficient, scalable engine, where one can concentrate on the ranking algorithm alone and does not have to deal with the crawling and database performance issues. Thus allowing more competing offers for search engines, which I believe is a good thing.

Second, secrecy does not protect against exploitation, as the current situation with search engines shows pretty well. You can also see this in the multitude of successful attacks against Microsoft's OS, despite the fact that its source code is secrete. We live in a world where reverse engineering is taught in colleges and is part of learning and the free flow of information.

In the security community, a encryption algorithm is only considered secure, if its working is well known and public knowledge. I think we should apply the same principles towards search engines. I believe we then can come up with methods and algorithms that will be exploitation reduced (or free).

Do I know a solution for this? No! But I believe Nutch gives everyone who has an idea, a platform to try it out. Nutch could be a stepping stone towards the solution.

What are your thoughts?

K<o>