Evaluating digital libraries

Conducting a comprehensive evaluation of a digital library requires a "triangulation" approach including multiple models, procedures, and tools. Carrying out valid evaluations of digital libraries in a timely and efficient manner is the focus of this tutorial. Why is evaluation of digital libraries so important? Each year sees the introduction of new digital libraries promoted as valuable resources for education and other needs. Yet systematic evaluation of the implementation and efficacy of these digital library systems is often lacking. This tutorial is specifically designed to establish evaluation as a key strategy throughout the design, development, and implementation of digital libraries. The tutorial focuses on a decision-oriented model for evaluating digital libraries using multiple methods such as: service evaluation, usability evaluation, information retrieval, biometrics evaluation, transaction log analysis survey methods, interviews and focus groups, observations, and experimental methods. Participants in this tutorial learn how to implement models and procedures for evaluating digital libraries at all levels of education. The tutorial includes presentations with actual case studies that are focused on a variety of digital library evaluation strategies. Participants also receive a copy of Evaluating Digital Libraries: A User-Friendly Guide

[1]  Richard E. Clark,et al.  Learning from media : arguments, analysis, and evidence , 2001 .

[2]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[3]  Mark W. Lipsey,et al.  Evaluation: A Systematic Approach , 1979 .

[4]  R. Westrum.  The Social Construction of Technological Systems , 1989 .

[5]  Jane Reid,et al.  A Task-Oriented Non-Interactive Evaluation Methodology for Information Retrieval Systems , 2000, Information Retrieval.

[6]  Janice Redish,et al.  User and task analysis for interface design , 1998 .

[7]  Anita Coleman,et al.  The Design and Evaluation of Interactivities in a Digital Library , 2002, D Lib Mag..

[8]  Lynn Lyons Morris,et al.  How to design a program evaluation , 1978 .

[9]  Gary Marchionini,et al.  The People in Digital Libraries: Multifaceted Approaches to Assessing Needs and Impact , 1999 .

[10]  Konstantinos I. Diamantaras,et al.  Experimental User-Centered Evaluation of an Open Hypermedia System and Web Information Seeking Environments , 2006, J. Digit. Inf..

[11]  Tefko Saracevic,et al.  Digital Library Evaluation: Toward Evolution of Concepts , 2000, Libr. Trends.

[12]  Karen E. Fisher,et al.  Determining How Libraries and Librarians Help , 2003, Libr. Trends.

[13]  Thomas C. Reeves,et al.  Interactive Learning Systems Evaluation , 2003 .

[14]  Massimo Melucci,et al.  An Evaluation of Automatically Constructed Hypertexts for Information Retrieval , 1999, Information Retrieval.

[15]  Eric Deeson,et al.  Interactive learning systems evaluation , 2004, Br. J. Educ. Technol..

[16]  Colin Sharp.  Qualitative Research and Evaluation Methods (3rd ed.) , 2003 .

[17]  D. Campbell,et al.  EXPERIMENTAL AND QUASI-EXPERIMENT Al DESIGNS FOR RESEARCH , 2012 .

[18]  Sally Jo Cunningham,et al.  A transaction log analysis of a digital library , 2000, International Journal on Digital Libraries.

[19]  Tefko Saracevic,et al.  A Survey of Digital Library Education. , 2001 .

[20]  Gary Marchionini,et al.  How fast is too fast? evaluating fast forward surrogates for digital video , 2003, 2003 Joint Conference on Digital Libraries, 2003. Proceedings..

[21]  M. Patton.  Qualitative research and evaluation methods , 1980 .

[22]  Tefko Saracevic,et al.  Evaluation of evaluation in information retrieval , 1995, SIGIR '95.

[23]  Randolph G. Bias,et al.  The pluralistic usability walkthrough: coordinated empathies , 1994 .

[24]  P. Haggard.  Look and feel , 2001, Trends in Cognitive Sciences.

[25]  Gary Marchionini,et al.  Evaluating Digital Libraries: A Longitudinal and Multifaceted View , 2000, Libr. Trends.

[26]  Elaina Norlin.  Reference Evaluation: A Three-Step Approach—Surveys, Unobtrusive Observations, and Focus Groups , 2000 .

[27]  Matthew B. Miles,et al.  Qualitative Data Analysis: An Expanded Sourcebook , 1994 .

[28]  John L. Casti,et al.  Paradigms Lost: Images of Man in the Mirror of Science , 1991 .

[29]  Wiebe E. Bijker,et al.  Of Bicycles, Bakelites, and Bulbs: Toward a Theory of Sociotechnical Change , 1995 .

[30]  Etienne Wenger,et al.  Communities of Practice: Learning, Meaning, and Identity , 1998 .

[31]  Arlene Fink,et al.  The Survey Handbook , 1995 .

[32]  Danny P. Wallace,et al.  Library Evaluation: A Casebook and Can-Do Guide , 2001 .

[33]  Richard A. Krueger,et al.  Focus groups : a practical guide for applied research / by Richard A. Krueger , 1989 .

[34]  Ben Shneiderman,et al.  Perspective-based Usability Inspection: An Empirical Validation of Efficacy , 1999, Empirical Software Engineering.

[35]  Jakob Nielsen,et al.  Designing web usability , 1999 .

[36]  Jakob Nielsen,et al.  Usability inspection methods , 1994, CHI 95 Conference Companion.

[37]  Stephen P. Harter,et al.  Evaluation of information retrieval systems : Approaches, issues, and methods , 1997 .

[38]  P. Hernon.  Quality: New Directions in the Research. , 2002 .

[39]  William G. Cochran,et al.  Experimental Designs, 2nd Edition , 1950 .

[40]  P. Hutchings,et al.  Opening Lines: Approaches to the Scholarship of Teaching and Learning. , 2000 .

[41]  G. Sayeed Choudhury,et al.  A Framework for Evaluating Digital Library Services , 2002, D Lib Mag..

[42]  Michael J. Kahn,et al.  Formal usability inspections , 1994 .

[43]  Linda C. Smith,et al.  Known-item search , 2006 .

[44]  Beth Sandore,et al.  Transaction log analysis , 1993 .

[45]  Mei-Mei Wu,et al.  Reflections on Information Retrieval Evaluation , 1999 .

[46]  Michael Quinn Patton,et al.  Utilization-focused evaluation: The new century text (3rd ed.). , 1997 .

[47]  Thomas C. Reeves,et al.  Evaluating Digital Libraries: A User-Friendly Guide , 2003 .

[48]  Geri Gay,et al.  Concept mapping: an innovative approach to digital library design and evaluation , 1995, SIGO.

[49]  Nancy A. Vanhouse,et al.  Digital Library Use: Social Practice in Design and Evaluation , 2003 .

[50]  Carol Tenopir,et al.  Use and Users of Electronic Library Resources: An Overview and Analysis of Recent Research Studies , 2003 .

[51]  Elaina Norlin,et al.  Usability testing for library web sites : a hands-on guide , 2002 .

[52]  Ann Blandford,et al.  An investigation into the application of Claims Analysis to evaluate usability of a digital library interface , 2002 .

[53]  Dennis Gabor,et al.  Inventing the Future , 1963 .

[54]  Martha Kyrillidou,et al.  Emerging Tools for Evaluating Digital Library Services: Conceptual Adaptations of LibQUAL+ and CAPM , 2003, J. Digit. Inf..

[55]  Joan M. Cherry,et al.  Studying digital library users over time: a follow-up survey of Early Canadiana Online , 2002, Inf. Res..

[56]  Huey-tsyh Chen.  Theory-driven evaluations , 1990 .

[57]  Mary F. Casserly,et al.  Usage and Usability Assessment: Library Practices and Concerns , 2002 .

[58]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[59]  Y. Lincoln,et al.  Scientific Research in Education , 2004 .

[60]  Jonathan Furner,et al.  Scholarly communication and bibliometrics , 2005, Annu. Rev. Inf. Sci. Technol..

[61]  Brewster Kahle,et al.  Public Access to Digital Material , 2001, D Lib Mag..

[62]  Amy Friedlander,et al.  D-Lib Magazine: Publishing as the Honest Broker , 1998 .

[63]  Edward A. Fox,et al.  Digital libraries , 1995, CACM.

[64]  Joan C. Durrance,et al.  Public use of digital community information sstems: findings from a recent study with implications for system design , 2001, JCDL '01.

[65]  Vannevar Bush,et al.  As we may think , 1945, INTR.

[66]  Melvin M. Mark,et al.  Multiple Methods in Program Evaluation , 1987 .

[67]  Arlene Fink,et al.  The Survey Kit , 1995 .

[68]  Joseph J. Schwab.  The Practical: A Language for Curriculum , 1969, The School Review.

[69]  Johan Bollen,et al.  Evaluation of Digital Library Impact and User Communities by Analysis of Usage Patterns , 2002, D Lib Mag..

[70]  W. Shadish,et al.  Foundations of Program Evaluation: Theories of Practice , 1990 .

[71]  Tom Peters,et al.  The history and development of transaction log analysis , 1993 .

[72]  Martha Kyrillidou,et al.  From input and output measures to quality and outcome measures, or, from the user in the life of the library to the library in the life of the user , 2002 .