Users' relevance criteria for evaluating answers in a social Q&A site

This study examines the criteria questioners use to select the best answers in a social Q&A site (Yahoo! Answers) within the theoretical framework of relevance research. A social Q&A site is a novel environment where people voluntarily ask and answer questions. InYahoo! Answers, the questioner selects the answer that best satisfies his or her question and leaves comments on it. Under the assumption that the comments reflect the reasons why questioners select particular answers as the best, this study analyzed 2,140 comments collected from Yahoo! Answers during December 2007. The content analysis identified 23 individual relevance criteria in six classes: Content, Cognitive, Utility, Information Sources, Extrinsic, and Socioemotional. A major finding is that the selection criteria used in a social Q&A site have considerable overlap with many relevance criteria uncovered in previous relevance studies, but that the scope of socioemotional criteria has been expanded to include the social aspect of this environment. Another significant finding is that the relative importance of individual criteria varies according to topic categories. Socioemotional criteria are popular in discussion-oriented categories, content-oriented criteria in topic-oriented categories,and utility criteria in self-help categories. This study generalizes previous relevance studies to a new environment by going beyond an academic setting.

[1]  Tefko Saracevic.  Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance , 2007 .

[2]  T. Saracevic,et al.  Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance , 2007, J. Assoc. Inf. Sci. Technol..

[3]  Edie M. Rasmussen,et al.  Users' relevance criteria in image retrieval in American history , 2002, Inf. Process. Manag..

[4]  Jarkko Kari,et al.  User-defined relevance criteria in web searching , 2006, J. Documentation.

[5]  Stefano Mizzaro.  Relevance: the whole history , 1997 .

[6]  Pia Borlund,et al.  The concept of relevance in IR , 2003, J. Assoc. Inf. Sci. Technol..

[7]  Carol L. Barry,et al.  Users' Criteria for Relevance Evaluation: A Cross-situational Comparison , 1998, Inf. Process. Manag..

[8]  Sanghee Oh,et al.  Best-answer selection criteria in a social Q&A site from the user-oriented relevance perspective , 2008, ASIST.

[9]  Diane H. Sonnenwald,et al.  User perspectives on relevance criteria: A comparison among relevant, partially relevant, and not-relevant judgments , 2002, J. Assoc. Inf. Sci. Technol..

[10]  Carol L. Barry.  User-defined relevance criteria: an exploratory study , 1994 .

[11]  Peter Ingwersen,et al.  Dimensions of relevance , 2000, Inf. Process. Manag..

[12]  Linda Schamber.  Relevance and Information Behavior. , 1994 .

[13]  Michael B. Eisenberg,et al.  A re-examination of relevance: toward a dynamic, situational definition , 1990, Inf. Process. Manag..

[14]  Yunjie Calvin Xu,et al.  Relevance judgment in epistemic and hedonic information searches , 2007, J. Assoc. Inf. Sci. Technol..

[15]  Douglas W. Oard,et al.  Searching large collections of recorded speech: A preliminary study , 2005, ASIST.

[16]  Sheizaf Rafaeli,et al.  Predictors of answer quality in online Q&A sites , 2008, CHI.

[17]  Rich Gazan.  Specialists and synthesists in a question answering community , 2006, ASIST.

[18]  Lada A. Adamic,et al.  Knowledge sharing and yahoo answers: everyone knows something , 2008, WWW.

[19]  Gary Marchionini,et al.  Exploring users' video relevance criteria - A pilot study , 2005, ASIST.

[20]  H. Winefield,et al.  Breast and Prostate Cancer Online Discussion Boards , 2007, Journal of health psychology.

[21]  Joemon M. Jose,et al.  Searchers' criteria For assessing web pages , 2003, SIGIR '03.

[22]  Linda Schamber,et al.  Users' Criteria for Evaluation in a Multimedia Environment. , 1991 .

[23]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[24]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[25]  Kara Reuter.  Assessing aesthetic relevance: Children's book selection in a digital library , 2007 .