Wednesday, January 2, 2008

Wiki-inspired "transparent" search-engine

Share
Wikia Search is a new, wiki-inspired search-engine project that attempts to create a transparent set of ranking algorithms that fight spam and promote good stuff to the top. This is in contrast to Google, Yahoo, and other search engines, where the ranking algorithms are treated as trade secrets and high-risk tactics that have to be guarded from spammers.

The idea of a ranking algorithm is that it produces "good results" -- returns the best, most relevant results based on the user's search terms. We have a notion that the traditional search engine algorithm is "neutral" -- that it lacks an editorial bias and simply works to fulfill some mathematical destiny, embodying some Platonic ideal of "relevance." Compare this to an "inorganic" paid search result of the sort that Altavista used to sell.

But ranking algorithms are editorial: they embody the biases, hopes, beliefs and hypotheses of the programmers who write and design them. What's more, a tiny handful of search engines effectively control the prominence and viability of the majority of the information in the world.

And those search engines use secret ranking systems to systematically and secretly block enormous swaths of information on the grounds that it is spam, malware, or using deceptive "optimization" techniques. The list of block-ees is never published, nor are the criteria for blocking. This is done in the name of security, on the grounds that spammers and malware hackers are slowed down by the secrecy.

But "security through obscurity" is widely discredited in information security circles. Obscurity stops dumb attackers from getting through, but it lets the smart attackers clobber you because the smart defenders can't see how your system works and point out its flaws.

Seen in this light, it's positively bizarre: a few companies' secret editorial criteria are used to control what information we see, and those companies defend their secrecy in the name of security-through-obscurity? Yikes!

source: www.boingboing.net


Powered by ScribeFire.

No comments: