Evaluation of digital libraries: Criteria and problems from users' perspectives

Digital library research has developed over the past decade, but little has been done on the identification of evaluation criteria, especially from users' perspectives. This article identifies users' criteria and applies them to the evaluation of existing digital libraries. Forty-eight subjects were instructed to develop and justify a set of essential criteria for the evaluation of digital libraries. At the same time, they were requested to evaluate existing digital libraries by applying the criteria that they were developing. A compilation of criteria developed by participants show that usability and collection quality were the most important criteria for evaluating digital libraries. Service quality, system performance efficiency, and user opinion solicitation were also deemed essential criteria. The author further compares digital library evaluation criteria identified by users and researchers and applied in previous research. The article reveals problems in current digital library design and development, discusses suggestions for improving digital library design, and presents areas for further research.

[1]  Christine L. Borgman,et al.  What are Digital Libraries? Competing Visions , 1999, Inf. Process. Manag..

[2]  Tefko Saracevic,et al.  Digital Library Evaluation: Toward Evolution of Concepts , 2000, Libr. Trends.

[3]  Johan Bollen,et al.  Usage Analysis for the Identification of Research Trends in Digital Libraries , 2003, D Lib Mag..

[4]  Rich Gazan,et al.  Iterative Design and Evaluation of a Geographic Digital Library for University Students: A Case Study of the Alexandria Digital Earth Prototype (ADEPT) , 2001, ECDL.

[5]  David S. Carter,et al.  Unobtrusive Data Analysis of Digital Reference Questions and Service at the Internet Public Library: An Exploratory Study , 2000, Libr. Trends.

[6]  Gary Marchionini,et al.  Interfaces and Tools for the Library of Congress National Digital Library Program , 1998, Inf. Process. Manag..

[7]  Ana D. Cleveland,et al.  Outline and Preliminary Evaluation of the Classical Digital Library Model. , 1999 .

[8]  Louise T. Su.  Evaluation Measures for Interactive Information Retrieval , 1992, Inf. Process. Manag..

[9]  HongWeiyin,et al.  Understanding user acceptance of digital libraries , 2002 .

[10]  Panagiotis Georgiou,et al.  A user-oriented evaluation of digital libraries: case study the "electronic journals" service of the library and information service of the University of Patras, Greece , 2002, Aslib Proc..

[11]  Gary Marchionini,et al.  Evaluating Digital Libraries: A Longitudinal and Multifaceted View , 2000, Libr. Trends.

[12]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[13]  B. Schneirdeman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[14]  Kathleen M. Heim.  If you want to evaluate your library , 1990 .

[15]  Paul B. Kantor,et al.  Studying the value of library and information services. Part II. Methodology and taxonomy , 1997 .

[16]  Kar Yan Tam,et al.  Understanding User Acceptance of Digital Libraries: What are the Roles of Interface Characteristics, Organizational Context, and Individual Differences? , 2002, Int. J. Hum. Comput. Stud..

[17]  Nancy A. Van House,et al.  User-Centered Iterative Design for Digital Libraries: The Cypress Experience , 1996, D Lib Mag..

[18]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[19]  Ann Peterson Bishop,et al.  Digital libraries: situating use in changing information infrastructure , 2000 .

[20]  John Carlo Bertot,et al.  Outcomes Assessment in the Networked Environment: Research Questions, Issues, Considerations, and Moving Forward , 2003, Libr. Trends.

[21]  Ahmad Rafee Che Kassim,et al.  Designing, implementing, and evaluating an educational digital library resource , 2003, Online Inf. Rev..

[22]  Tefko Saracevic,et al.  Challenges for Digital Library Evaluation. , 2000 .

[23]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[24]  Douglas L. Zweizig.  If You Want to Evaluate Your Library (2nd ed.), by F. W. Lancaster , 1995, J. Am. Soc. Inf. Sci..

[25]  Gobinda G. Chowdhury,et al.  Introduction to Digital Libraries , 2002 .

[26]  Martha Kyrillidou,et al.  Emerging Tools for Evaluating Digital Library Services: Conceptual Adaptations of LibQUAL+ and CAPM , 2003, J. Digit. Inf..

[27]  Joan M. Cherry,et al.  Studying digital library users over time: a follow-up survey of Early Canadiana Online , 2002, Inf. Res..

[28]  Terence R. Smith,et al.  Alexandria digital library: user evaluation studies and system design , 2000 .