Contrasting Search as a Learning Activity with Instructor-designed Learning

The field of Search as Learning addresses questions surrounding human learning during the search process. Existing research has largely focused on observing how users with learning-oriented information needs behave and interact with search engines. What is not yet quantified is the extent to which search is a viable learning activity compared to instructor-designed learning. Can a search session be as effective as a lecture video - our instructor-designed learning artefact - or learning? To answer this question, we designed a user study that pits instructor-designed learning (a short high-quality video lecture as commonly found in online learning platforms) against three instances of search, specifically (i) single-user search, (ii) search as a support tool for instructor-designed learning, and, (iii) collaborative search. We measured the learning gains of 151 study participants in a vocabulary learning task and report three main results: (i) lecture video watching yields up to 24% higher learning gains than single-user search, (ii) collaborative search for learning does not lead to increased learning, and (iii) lecture video watching supported by search leads up to a 41% improvement in learning gains over instructor-designed learning without a subsequent search phase.

[1]  A. Acquisti,et al.  Beyond the Turk: Alternative Platforms for Crowdsourcing Behavioral Research , 2016 .

[2]  Abdigani Diriye,et al.  Querium: A Session-Based Collaborative Search System , 2012, ECIR.

[3]  Ryen W. White,et al.  Supporting Complex Search Tasks , 2014, CIKM.

[4]  Laure Soulier,et al.  On the Collaboration Support in Information Retrieval , 2017, ACM Comput. Surv..

[5]  Maarten de Rijke,et al.  Aggregated search interface preferences in multi-session search tasks , 2013, SIGIR.

[6]  Arthur R. Taylor,et al.  A study of the information search behaviour of the millennial generation , 2012, Inf. Res..

[7]  Barrie Gunter,et al.  The Google generation: the information behaviour of the researcher of the future , 2008, Aslib Proc..

[8]  Ryen W. White,et al.  Lessons from the journey: a query log analysis of within-session learning , 2014, WSDM.

[9]  Dorota Glowacka,et al.  SciNet: Interactive Intent Modeling for Information Discovery , 2015, SIGIR.

[10]  Meredith Ringel Morris,et al.  Collaborative search revisited , 2013, CSCW.

[11]  Leif Azzopardi,et al.  Pienapple search: an integrated search interface to support finding, refinding and sharing , 2016, ASIST.

[12]  Han Woo Park,et al.  Convenience or credibility? A study of college student online research behaviors , 2011, Internet High. Educ..

[13]  Peter Williams,et al.  Google Generation II: web behaviour experiments with the BBC , 2011, Aslib Proc..

[14]  Peter Bailey,et al.  User task understanding: a web search engine perspective , 2012 .

[15]  M. Wesche,et al.  Assessing Second Language Vocabulary Knowledge: Depth Versus Breadth. , 1996 .

[16]  Kevyn Collins-Thompson,et al.  Optimizing search results for human learning goals , 2017, Information Retrieval Journal.

[17]  Meredith Ringel Morris,et al.  CoSearch: a system for co-located collaborative web search , 2008, CHI.

[18]  Gary Marchionini,et al.  Synthesis Lectures on Information Concepts, Retrieval, and Services , 2009 .

[19]  Edgar Dale,et al.  Vocabulary Measurement: Techniques and Major Findings , 2016 .

[20]  Max L. Wilson,et al.  A comparison of techniques for measuring sensemaking and learning within participant-generated summaries , 2013, J. Assoc. Inf. Sci. Technol..

[21]  Felipe Moraes,et al.  SearchX: Empowering Collaborative Search Research , 2018, SIGIR.

[22]  Urs Gasser,et al.  Youth and Digital Media: From Credibility to Information Quality , 2012 .

[23]  Ryen W. White,et al.  Exploratory Search: Beyond the Query-Response Paradigm , 2009, Exploratory Search: Beyond the Query-Response Paradigm.

[24]  Philip J. Guo,et al.  How video production affects student engagement: an empirical study of MOOC videos , 2014, L@S.

[25]  Dorota Glowacka,et al.  Beyond Relevance: Adapting Exploration/Exploitation in Information Retrieval , 2016, IUI.

[26]  Abdigani Diriye,et al.  The future is in the past: designing for exploratory search , 2012, IIiX.

[27]  Kevyn Collins-Thompson,et al.  Search as Learning (Dagstuhl Seminar 17092) , 2017, Dagstuhl Reports.

[28]  Nicholas J. Belkin,et al.  Predicting users' domain knowledge from search behaviors , 2011, SIGIR.

[29]  Kevyn Collins-Thompson,et al.  Retrieval Algorithms Optimized for Human Learning , 2017, SIGIR.

[30]  Kevyn Collins-Thompson,et al.  Assessing Learning Outcomes in Web Search: A Comparison of Tasks and Query Strategies , 2016, CHIIR.

[31]  Gary Marchionini,et al.  Exploratory search , 2006, Commun. ACM.

[32]  Chirag Shah,et al.  Role-based results redistribution for collaborative information retrieval , 2010, Inf. Process. Manag..

[33]  Edward Cutrell,et al.  What are you looking for?: an eye-tracking study of information usage in web search , 2007, CHI.

[34]  James Allan,et al.  Frontiers, challenges, and opportunities for information retrieval: Report from SWIRL 2012 the second strategic workshop on information retrieval in Lorne , 2012, SIGF.

[35]  Robert G. Capra,et al.  Influence of training and stage of search on gaze behavior in a library catalog faceted search interface , 2012, J. Assoc. Inf. Sci. Technol..

[36]  Ryen W. White,et al.  Characterizing the influence of domain expertise on web search behavior , 2009, WSDM '09.

[37]  Maribeth Back,et al.  A Taxonomy of Collaboration in Online Information Seeking , 2009, ArXiv.

[38]  Gary Marchionini,et al.  Design and evaluation of a system to support collaborative search , 2012, ASIST.

[39]  Jacek Gwizdka,et al.  Inferring user knowledge level from eye movement patterns , 2013, Inf. Process. Manag..

[40]  Ryen W. White,et al.  Slow Search: Information Retrieval without Time Constraints , 2013, HCIR '13.

[41]  Meredith Ringel Morris,et al.  CoSense: enhancing sensemaking for collaborative web search , 2009, CHI.

[42]  Dorota Glowacka,et al.  Is exploratory search different? A comparison of information search behavior for exploratory and lookup tasks , 2016, J. Assoc. Inf. Sci. Technol..

[43]  Eric Horvitz,et al.  SearchTogether: an interface for collaborative web search , 2007, UIST.

[44]  Gary Marchionini,et al.  Report on ACM SIGIR 2006 workshop on evaluating exploratory search systems , 2006, SIGF.

[45]  Nicholas J. Belkin,et al.  Second Workshop on Supporting Complex Search Tasks , 2017, CHIIR.

[46]  Benjamin S. Bloom,et al.  A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives , 2000 .

[47]  Marco A. Bravo,et al.  Contemporary Classroom / Vocabulary Assessment / for Content Areas , 2010 .

[48]  Chirag Shah,et al.  Coagmento: A system for supporting collaborative information seeking , 2011, ASIST.