Social Q&A and virtual reference - comparing apples and oranges with the help of experts and users

Online question-answering (Q&A) services are becoming increasingly popular among information seekers. We divide them into two categories, social Q&A (SQA) and virtual reference (VR), and examine how experts (librarians) and end users (students) evaluate information within both categories. To accomplish this, we first performed an extensive literature review and compiled a list of the aspects found to contribute to a “good” answer. These aspects were divided among three high-level concepts: relevance, quality, and satisfaction. We then interviewed both experts and users, asking them first to reflect on their online Q&A experiences and then comment on our list of aspects. These interviews uncovered two main disparities. One disparity was found between users’ expectations with these services and how information was actually delivered among them, and the other disparity between the perceptions of users and experts with regard to the aforementioned three characteristics of relevance, quality, and satisfaction. Using qualitative analyses of both the interviews and relevant literature, we suggest ways to create better hybrid solutions for online Q&A and to bridge the gap between experts’ and users’ understandings of relevance, quality, and satisfaction, as well as the perceived importance of each in contributing to a good answer. © 2012 Wiley Periodicals, Inc.

[1]  Peiling Wang,et al.  Consolidating user relevance criteria: A meta-ethnography of empirical studies , 2006, ASIST.

[2]  Sanghee Oh,et al.  Best-answer selection criteria in a social Q&A site from the user-oriented relevance perspective , 2008, ASIST.

[3]  Jeffrey Pomerantz,et al.  Motivations and uses: Evaluating virtual reference service from the users' perspective , 2006 .

[4]  Peiling Wang,et al.  A cognitive model of document use during a research project. Study I. document selection , 1998 .

[5]  Wen-Lian Hsu,et al.  journal homepage: www.elsevier.com/locate/im , 2022 .

[6]  Joyce Ward,et al.  Best Practices in Chat Reference Used by Florida's Ask a Librarian Virtual Reference Librarians , 2009 .

[7]  Terri Pedersen Summey,et al.  From IM to Collaboration , 2007 .

[8]  R. David Lankes,et al.  Statistics, Measures and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Protocols , 2004 .

[9]  Chirag Shah,et al.  Faster, better, or both? Looking at both sides of online question-answering coin , 2011, ASIST.

[10]  Diane H. Sonnenwald,et al.  User perspectives on relevance criteria: A comparison among relevant, partially relevant, and not-relevant judgments , 2002, J. Assoc. Inf. Sci. Technol..

[11]  Colleen Cool,et al.  Information Retrieval as Symbolic Interaction: Examples from Humanities Scholars. , 1993 .

[12]  Chirag Shah.  Measuring effectiveness and user satisfaction in Yahoo! Answers , 2011, First Monday.

[13]  Marianne Foley,et al.  Instant Messaging Reference in an Academic Library: A Case Study , 2002 .

[14]  Robert Slater.  Reference Desk Notes , 2006 .

[15]  S. Hirsh.  Children's Relevance Criteria and Information Seeking on Electronic Resources , 1999, J. Am. Soc. Inf. Sci..

[16]  Birger Hjørland.  The foundation of the concept of relevance , 2010 .

[17]  Scott Norwood,et al.  Applying RUSA Guidelines in the Analysis of Chat Reference Transcripts , 2006 .

[18]  W. Bruce Croft,et al.  A framework to predict the quality of answers with non-textual features , 2006, SIGIR.

[19]  Pnina Shachaf,et al.  A structuration approach to online communities of practice: The case of Q&A communities , 2010, AMCIS 2010.

[20]  Rich Gazan,et al.  Microcollaborations in a social Q&A community , 2010, Inf. Process. Manag..

[21]  Chirag Shah,et al.  The use of information sources by internet users in answering questions , 2008, ASIST.

[22]  Lorri Mon,et al.  On the Boundaries of Reference Services: Questioning and Library 2.0 , 2009 .

[23]  J. Oh,et al.  Research agenda for social Q&A , 2009 .

[24]  Jeffrey Pomerantz,et al.  Peer Review of Chat Reference Transcripts: Approaches and Strategies , 2006 .

[25]  Chirag Shah,et al.  Exploring Characteristics and Effects of User Participation in Online Social Q&A Sites , 2008, First Monday.

[26]  Steve Smithson,et al.  Information Retrieval Evaluation in Practice: A Case Study Approach , 1994, Inf. Process. Manag..

[27]  Rich Gazan,et al.  Seekers, sloths and social reference: Homework questions submitted to a question-answering community , 2007, New Rev. Hypermedia Multim..

[28]  Rong Tang,et al.  Use of relevance criteria across stages of document evaluation: on the complementarity of experimental and naturalistic studies , 2001 .

[29]  Pnina Shachaf,et al.  Online Social Reference: A Research Agenda Through a STIN Framework , 2009 .

[30]  Celia Hales Mabry,et al.  The Reference Interview as Partnership: An Examination of Librarian, Library User, and Social Interaction , 2004 .

[31]  Jeffrey Pomerantz,et al.  A conceptual framework and open research questions for chat-based reference service , 2005, J. Assoc. Inf. Sci. Technol..

[32]  Eileen G. Abels,et al.  A Model of the Reference and Information Service Process , 2011 .

[33]  Nicholas J. Belkin,et al.  Relationships between categories of relevance criteria and stage in task completion , 2007, Inf. Process. Manag..

[34]  David S. Carter,et al.  Unobtrusive Data Analysis of Digital Reference Questions and Service at the Internet Public Library: An Exploratory Study , 2000, Libr. Trends.

[35]  Marilyn Ochoa,et al.  Application of Reference Guidelines in Chat Reference Interactions , 2007 .

[36]  T. Park.  The Nature of Relevance in Information Retrieval: An Empirical Study , 1993, The Library Quarterly.

[37]  Lada A. Adamic,et al.  Knowledge sharing and yahoo answers: everyone knows something , 2008, WWW.

[38]  Pia Borlund,et al.  The concept of relevance in IR , 2003, J. Assoc. Inf. Sci. Technol..

[39]  R. David Lankes,et al.  The current state of digital reference: validation of a general digital reference model through a survey of digital reference services , 2004, Inf. Process. Manag..

[40]  Patrick Wilson,et al.  The Cognitive and the Social@@@The Social Construction of Science: A Comparative Study of Goal Direction Research Evolution, and Legitimation.@@@Second-Hand Knowledge: An Inquiry into Cognitive Authority. , 1983 .

[41]  Carol L. Barry.  User-defined relevance criteria: an exploratory study , 1994 .

[42]  Jeffrey Pomerantz.  A conceptual framework and open research questions for chat-based reference service: Research Articles , 2005 .

[43]  Linda Schamber.  Users' criteria for evaluation in multimedia information seeking and use situations , 1991 .

[44]  Carol L. Barry,et al.  Users' Criteria for Relevance Evaluation: A Cross-situational Comparison , 1998, Inf. Process. Manag..

[45]  Mark S. Ackerman,et al.  Expertise networks in online communities: structure and algorithms , 2007, WWW '07.

[46]  Yunjie Xu,et al.  Relevance judgment: What do information users consider beyond topicality? , 2006 .

[47]  Evgeniy Gabrilovich,et al.  Predicting web searcher satisfaction with existing community-based answers , 2011, SIGIR.

[48]  Harry Bruce.  A cognitive view of the situational dynamism of user-centered relevance estimation , 1994 .

[49]  Stephanie Willen Brown,et al.  The Reference Interview: Theories and Practice , 2008 .

[50]  N. Roberts,et al.  Value-added processes in information systems , 1986 .

[51]  Sanghee Oh,et al.  The relationships between motivations and answering strategies: An exploratory review of health answerers' behaviors in Yahoo! answers , 2011, ASIST.

[52]  Soojung Kim,et al.  Users' relevance criteria for evaluating answers in a social Q&A site , 2009 .

[53]  Pnina Shachaf,et al.  Virtual reference service evaluation: Adherence to RUSA behavioral guidelines and IFLA digital reference guidelines , 2008 .

[54]  Thomas J. Froehlich,et al.  Relevance reconsidered—towards an agenda for the 21st century: introduction to special topic issue on relevance research , 1994 .

[55]  Sanghee Oh,et al.  Evaluating health answers in a social Q&A site , 2009, ASIST.

[56]  Bruce Stoffel,et al.  Why Isn't Our Chat Reference Used More? Finding of Focus Group Discussions with Undergraduate Students , 2008 .

[57]  Tefko Saracevic.  Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance , 2007 .

[58]  Sheizaf Rafaeli,et al.  Predictors of answer quality in online Q&A sites , 2008, CHI.

[59]  Rich Gazan.  Specialists and synthesists in a question answering community , 2006, ASIST.

[60]  Eugene Agichtein,et al.  Predicting information seeker satisfaction in community question answering , 2008, SIGIR '08.

[61]  George Kingsley Zipf,et al.  Human behavior and the principle of least effort , 1949 .

[62]  Pnina Shachaf.  Social reference: Toward a unifying theory , 2010 .

[63]  Alex Rolfe,et al.  Ask-an-expert services analysis , 2001, J. Assoc. Inf. Sci. Technol..

[64]  James Surowiecki.  The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations Doubleday Books. , 2004 .

[65]  Daphne R. Raban,et al.  User-Centered Evaluation of Information: A Research Challenge , 2007, Internet Res..

[66]  Jeffrey Pomerantz,et al.  Evaluating and predicting answer quality in community QA , 2010, SIGIR.

[67]  Birger Hjørland.  The foundation of the concept of relevance , 2009 .

[68]  Christopher Peter Lueg,et al.  Querying information systems or interacting with intermediaries? Towards understanding the informational capacity of online communities , 2008, ASIST.

[69]  Rich Gazan,et al.  Social Q&A , 2011, J. Assoc. Inf. Sci. Technol..

[70]  Marilyn Domas White,et al.  Evaluation of Chat Reference Service Quality: Pilot Study , 2003, D Lib Mag..

[71]  Linda Schamber.  Relevance and Information Behavior. , 1994 .

[72]  Marie L. Radford.  Relational aspects of reference interactions : a qualitative investigation of the perceptions of users and librarians in the academic library , 1993 .