Ranking of Wikipedia articles in search engines revisited: Fair ranking for reasonable quality?

This paper aims to review the fiercely discussed question of whether the ranking of Wikipedia articles in search engines is justified by the quality of the articles. After an overview of current research on information quality in Wikipedia, a summary of the extended discussion on the quality of encyclopedic entries in general is given. On this basis, a heuristic method for evaluating Wikipedia entries is developed and applied to Wikipedia articles that scored highly in a search engine retrieval effectiveness test and compared with the relevance judgment of jurors. In all search engines tested, Wikipedia results are unanimously judged better by the jurors than other results on the corresponding results position. Relevance judgments often roughly correspond with the results from the heuristic evaluation. Cases in which high relevance judgments are not in accordance with the comparatively low score from the heuristic evaluation are interpreted as an indicator of a high degree of trust in Wikipedia. One of the systemic shortcomings of Wikipedia lies in its necessarily incoherent user model. A further tuning of the suggested criteria catalog, for instance, the different weighing of the supplied criteria, could serve as a starting point for a user model differentiated evaluation of Wikipedia articles. Approved methods of quality evaluation of reference works are applied to Wikipedia articles and integrated with the question of search engine evaluation. © 2011 Wiley Periodicals, Inc.

[1]  Les Gasser,et al.  Assessing Information Quality of a Community-Based Encyclopedia , 2005, ICIQ.

[2]  Sean W. Smith,et al.  Quality in Internet Collective Goods : Zealots and Good Samaritans in the Case of Wikipedia , 2005 .

[3]  William A. Katz,et al.  Introduction to reference work , 1969 .

[4]  Winfried Gödert.  Multimedia-Enzyklopädien auf CD-ROM - eine vergleichende Analyse von Allgemeinenzyklopädien , 1994 .

[5]  Ulrike Spree,et al.  Das Streben nach Wissen : eine vergleichende Gattungsgeschichte der populären Enzyklopädie in Deutschland und Großbritannien im 19. Jahrhundert , 2000 .

[6]  Marios Poulos,et al.  Evaluating authoritative sources using social networks: an insight from Wikipedia , 2006, Online Inf. Rev..

[7]  Jakob Nielsen,et al.  Usability inspection methods , 1994, CHI 95 Conference Companion.

[8]  Brendan Luyt,et al.  Improving Wikipedia's credibility: References and citations in a sample of history articles , 2010, J. Assoc. Inf. Sci. Technol..

[9]  Don Fallis,et al.  Toward an epistemology of Wikipedia , 2008, J. Assoc. Inf. Sci. Technol..

[10]  J. Giles.  Internet encyclopaedias go head to head , 2005, Nature.

[11]  Patrick Danowski,et al.  Wikipedia und Normdateien: Wege der Vernetzung am Beispiel der Kooperation mit der Personennamendatei , 2007 .

[12]  T. Saracevic,et al.  Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance , 2007, J. Assoc. Inf. Sci. Technol..

[13]  Oded Nov,et al.  What motivates Wikipedians? , 2007, CACM.

[14]  Gerald Benoît,et al.  Link analysis: An information science approach , 2006, J. Assoc. Inf. Sci. Technol..

[15]  Alex Soojung-Kim Pang.  The Work of the Encyclopedia in the Age of Electronic Reproduction , 1998, First Monday.

[16]  Besiki Stvilia.  A workbench for information quality evaluation , 2008, JCDL '08.

[17]  M. Makary,et al.  Wiki-Surgery? Internal validity of Wikipedia as a medical and surgical reference , 2007 .

[18]  Tefko Saracevic.  Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance , 2007 .

[19]  Sook Lim,et al.  How and why do college students use Wikipedia? , 2009, J. Assoc. Inf. Sci. Technol..

[20]  Jakob Voß,et al.  Collaborative thesaurus tagging the Wikipedia way , 2006, ArXiv.

[21]  Norbert Bachleitner,et al.  Ulrike Spree, Das Streben nach Wissen. Eine vergleichende Gattungsgeschichte der populären Enzyklopädie in Deutschland und Großbritannien im 19. Jahrhundert. 2000 , 2001 .

[22]  Sean W. Smith,et al.  The Quality of Open Source Production: Zealots and Good Samaritans in the Case of Wikipedia , 2007 .

[23]  Tefko Saracevic,et al.  Relevance : A Review of the Literature and a Framework for Thinking on the Notion in Information Science . Part III : Behavior and Effects of Relevance , 1976 .

[24]  Stefano Mizzaro,et al.  Relevance: The Whole History , 1997, J. Am. Soc. Inf. Sci..

[25]  Linda C. Smith,et al.  Reference and information services : an introduction , 1995 .

[26]  I. Mühlhauser,et al.  Sind medizinische und Gesundheitsinformationen auf den Internetseiten von Wikipedia evidenzbasiert? – Eine Inhaltsanalyse ☆ , 2008 .

[27]  R. Gottlieb,et al.  The New Yorker , 1987 .

[28]  Tom McArthur,et al.  Worlds of reference : lexicography, learning, and language from the clay tablet to the computer , 1986 .

[29]  Dirk Lewandowski,et al.  What Users See - Structures in Search Engine Results Pages , 2009, Inf. Sci..

[30]  I. Mühlhauser,et al.  [Does WIKIPEDIA provide evidence-based health care information? A content analysis]. , 2008, Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen.

[31]  F. A. Brockhaus Wiesbaden.  Der Grosse Brockhaus , 1952 .

[32]  Rainer Hammwöhner,et al.  Qualität der Wikipedia - Eine vergleichende Studie , 2007, ISI.

[33]  Rainer Hammwöhner.  Qualitätsaspekte der Wikipedia , 2007 .

[34]  Patrick Danowski,et al.  Das Wissen der Welt - Die Wikipedia , 2005 .

[35]  Don Fallis,et al.  Toward an epistemology of Wikipedia , 2008 .

[36]  Scott Nicholson,et al.  How much of it is real? Analysis of paid placement in Web search engine results , 2006, J. Assoc. Inf. Sci. Technol..

[37]  Frank Schulenburg,et al.  Der „McDonald's der Informationen“? Ein Blick hinter die Kulissen des kollaborativen Wissensmanagements in der deutschsprachigen Wikipedia , 2007 .

[38]  Pia Borlund,et al.  The concept of relevance in IR , 2003, J. Assoc. Inf. Sci. Technol..

[39]  Dirk Lewandowski,et al.  The Retrieval Effectiveness of Web Search Engines: Considering Results Descriptions , 2008, J. Documentation.

[40]  Alastair G. Smith,et al.  Testing the Surf: Criteria for Evaluating Internet Information Resources , 1997 .

[41]  Lucy Holman Rector.  Comparison of Wikipedia and other encyclopedias for accuracy, breadth, and depth in historical articles , 2008 .

[42]  William A. Katz.  Basic information services , 2002 .

[43]  Kenneth F. Kister.  Kister's Best Encyclopedias: A Comparative Guide to General and Specialized Encyclopedias , 1992 .

[44]  Thomas Chesney,et al.  An empirical examination of Wikipedia's credibility , 2006, First Monday.

[45]  Jutta Haider,et al.  Beyond the Legacy of the Enlightenment? Online Encyclopaedias as Digital Heterotopias , 2010, First Monday.