Barry Schwartz of SE Roundtable had an idea to put the relevancy of the major search engines to the test by creating a “white labeled” search engine that would display the results from the different search engines and ask users to rate the relevance of each. (Click that link before Barry gets buried for borrowing six trademarks).
This idea has been criticized and praised in various places, but Barry was undaunted and actually implemented a white labeled “RustySearch” engine. Thus, the great Search Engine Relevancy Challenge of 2005 was born.
Danny Sullivan and others have pointed out that there really isn’t much distance between the major search engines, and early results appear to bear that out. Yahoo has taken an early lead with Google right behind, Ask Jeeves and MSN are 3rd and 4th. The difference between the #1 Yahoo (score 3.38 out of 5) and #4 MSN (score 3.07) is not much to speak of, especially since these results are based on 3,200 searches which were probably performed mostly by SEOs with a fairly narrow range of search terms.
So is the Relevancy Challenge really relevant? Maybe not, but it’s still interesting, and I hope that Barry will continue it for a while. What would be really interesting would be to see the score of the competing search engines for different search term lengths. This would help us see, for example, whether Ask Jeeves scores better on generic searches due to their topical algorithm, but less well on long search terms because of their smaller database.
