Best methods for evaluating educational impact: a comparison of the efficacy of commonly used measures of library instruction.

OBJECTIVES AND BACKGROUND Libraries are increasingly called upon to demonstrate student learning outcomes and the tangible benefits of library educational programs. This study reviewed and compared the efficacy of traditionally used measures for assessing library instruction, examining the benefits and drawbacks of assessment measures and exploring the extent to which knowledge, attitudes, and behaviors actually paralleled demonstrated skill levels. METHODS An overview of recent literature on the evaluation of information literacy education addressed these questions: (1) What evaluation measures are commonly used for evaluating library instruction? (2) What are the pros and cons of popular evaluation measures? (3) What are the relationships between measures of skills versus measures of attitudes and behavior? Research outcomes were used to identify relationships between measures of attitudes, behaviors, and skills, which are typically gathered via attitudinal surveys, written skills tests, or graded exercises. RESULTS AND CONCLUSIONS Results provide useful information about the efficacy of instructional evaluation methods, including showing significant disparities between attitudes, skills, and information usage behaviors. This information can be used by librarians to implement the most appropriate evaluation methods for measuring important variables that accurately demonstrate students' attitudes, behaviors, or skills.

[1]  Francesca Allegri.  Information Literacy: Revolution in the Library. , 1990 .

[2]  Hal Grossman,et al.  Assessing the assessment: how institutions administered, interpreted, and used SAILS , 2010 .

[3]  B M Wildemuth,et al.  End-user search behaviors and their relationship to search effectiveness. , 1995, Bulletin of the Medical Library Association.

[4]  Kimberly Pendell,et al.  Assess and Invest: Faculty Feedback on Library Tutorials , 2010, Coll. Res. Libr..

[5]  Barbara Blummer.  Assessing Patron Learning from an Online Library Tutorial , 2007 .

[6]  Lin Wu,et al.  Web Tutorials , 2006, Medical reference services quarterly.

[7]  Susan A. Ariew,et al.  Evaluating Instruction: Developing a Program that Supports the Teaching Librarian. , 2005 .

[8]  Robert Mackoy,et al.  Effects of library instruction on university students' satisfaction with the library : A longitudinal study , 1998 .

[9]  Craig Gibson,et al.  Assessment Essentials: Planning, Implementing, and Improving Assessment in Higher Education , 2000 .

[10]  Claudine SchWeber,et al.  Library Instruction for the Next Millennium , 2001 .

[11]  Megan Oakleaf.  Dangers and Opportunities: A Conceptual Map of Information Literacy Assessment Approaches , 2008 .

[12]  Wen-Hua Ren,et al.  Library instruction and college student self-efficacy in electronic information searching , 2000 .

[13]  Shannon M. Staley,et al.  Standardised library instruction assessment: an institution-specific approach , 2010, Inf. Res..

[14]  Rui Wang,et al.  The Lasting Impact of a Library Credit Course , 2006 .

[15]  P. Maughan.  Assessing Information Literacy among Undergraduates: A Discussion of the Literature and the University of California-Berkeley Assessment Experience. , 2001 .

[16]  D. Edge.  Quantitative Measures of Communication in Science: A Critical Review , 1979, History of science; an annual review of literature, research and teaching.

[17]  Constance A. Mellon.  Process Not Product in Course-Integrated Instruction: A Generic Model of Library Research , 1984 .

[18]  Linda A. Suskie.  Assessing Student Learning: A Common Sense Guide , 2004 .

[19]  Karen D. Sobel,et al.  Updating Your Tool Belt , 2011 .

[20]  M L Pao,et al.  Factors affecting students' use of MEDLINE. , 1993, Computers and biomedical research, an international journal.

[21]  Anthony J. Onwuegbuzie,et al.  The relationship between library anxiety and learning styles among graduate students: Implications for library instruction , 1998 .

[22]  W. Rosenberg,et al.  Improving Searching Skills and Evidence Retrieval , 1998, Journal of the Royal College of Physicians of London.

[23]  Alice W. S. Chan,et al.  Course‐integrated learning outcomes for library database searching : three assessment points on the path of evidence , 2010 .

[24]  Elizabeth Blakesley Lindsay.  Distance Teaching: Comparing Two Online Information Literacy Courses. , 2004 .

[25]  E G Detlefsen,et al.  Implementation of a Web-based tutorial. , 1999, Bulletin of the Medical Library Association.

[26]  Barbara M. Wildemuth,et al.  End User Searching of Medline. Final Report. , 1993 .

[27]  T. Scott Plutchak.  On the Satisfied and Inept End User , 1989 .

[28]  S. Burrows,et al.  Evaluating medical student searches of MEDLINE for evidence-based information: process and application of results. , 1999, Bulletin of the Medical Library Association.

[29]  Bonnie J. M. Swoger.  Closing the assessment loop using pre‐ and post‐assessment , 2011 .

[30]  Stacy Nowicki.  Information Literacy and Critical Thinking in the Electronic Environment. , 1999 .

[31]  M L Pao,et al.  Effect of search experience on sustained MEDLINE usage by students , 1994, Academic medicine : journal of the Association of American Medical Colleges.

[32]  Joanna M. Burkhardt.  Assessing Library Skills: A First Step to Information Literacy , 2007 .

[33]  Raya Fidel,et al.  Moves in online searching , 1985 .

[34]  Robert S. Weisskirch,et al.  The effectiveness of project-specific information competence instruction , 2005 .

[35]  Cynthia M. Kisby.  Self-Assessed Learning and User Satisfaction in Regional Campus Libraries. , 2011 .

[36]  Elizabeth Figa,et al.  Faculty-Librarian Collaboration for Library Services in the Online Classroom: Student Evaluation Results and Recommended Practices for Implementation , 2009 .

[37]  John Crawford.  The use of electronic information services and information literacy , 2006, J. Libr. Inf. Sci..

[38]  Karen Bronshteyn,et al.  Perspectives on … Librarians as Writing Instructors: Using Paraphrasing Exercises to Teach Beginning Information Literacy Students , 2006 .

[39]  Caroline Cason Kristin Christy Ron Barratt,et al.  Collaboration is Key: Librarians and Composition Instructors Analyze Student Research and Writing , 2008 .

[40]  Sarah Polkinghorne,et al.  Research is a Verb: Exploring a New Information Literacy-Embedded Undergraduate , 2010 .

[41]  Jon R. Hufford.  What Are They Learning? Pre- and Post-Assessment Surveys for LIBR 1100, Introduction to Library Research , 2010, Coll. Res. Libr..

[42]  N. Elliot,et al.  Direct Assessment of Information Literacy Using Writing Portfolios. , 2007 .

[43]  Jenny Fafeita.  The Current Status of Teaching and Fostering Information Literacy in TAFE , 2006 .

[44]  Kate Zoellner,et al.  Continuing Assessment of Library Instruction to Undergraduates : A General Education Course Survey Research Project , 2008 .

[45]  Rachel Applegate,et al.  Models of User Satisfaction: Understanding False Positives , 1993 .

[46]  Li Zhang,et al.  The Efficacy of Computer-Assisted Instruction versus Face-to-Face Instruction in Academic Libraries: A Systematic Review , 2007 .

[47]  Katherine Schilling,et al.  An interactive web-based curriculum on evidence-based medicine: design and effectiveness. , 2006, Family medicine.

[48]  Jason Dupree,et al.  Assessing Learning, Critical Reflection, and Quality Educational Outcomes: The Critical Incident Questionnaire , 2008 .

[49]  Simon Canick.  Legal Research Assessment , 2009 .

[50]  Marilyn P. Whitmore,et al.  Bibliographic instruction : the second generation , 1988 .

[51]  Xin Wang,et al.  Computational Linguistic Assessment of Genre Differences Focusing on Text Cohesive Devices of Student Writing: Implications for Library Instruction , 2010 .

[52]  S. Wise,et al.  The Development and Validation of the Information Literacy Test , 2007 .

[53]  Philip J. Smith,et al.  Knowledge-Based Search Tactics , 1993, Inf. Process. Manag..

[54]  Patricia Senn Breivik.  Making the Most of Libraries in the Search for Academic Excellence , 1987 .

[55]  Megan Oakleaf,et al.  Are They Learning? Are We? Learning Outcomes and the Academic Library , 2011, The Library Quarterly.

[56]  Lara Cummings,et al.  If You Build It, Will They Learn? Assessing Online Information Literacy Tutorials. , 2006 .