A review of quality evaluation of digital libraries based on users’ perceptions

In the past two decades, the use of digital libraries (DLs) has grown significantly. Accordingly, questions about the utility, usability and cost of DLs have started to arise, and greater attention is being paid to the quality evaluation of this type of information system. Since DLs are destined to serve user communities, one of the main aspects to be considered in DL evaluation is the user’s opinion. The literature on this topic has produced a set of varied criteria to judge DLs from the user’s perspective, measuring instruments to elicit users’ opinions, and approaches to analyse the elicited data to conclude an evaluation. This paper provides a literature review of the quality evaluation of DLs based on users’ perceptions. Its main contribution is to bring together previously disparate streams of work to help shed light on this thriving area. In addition, the various studies are discussed, and some challenges to be faced in the future are proposed.

[1]  Hong Xie,et al.  Users' evaluation of digital libraries (DLs): Their uses, their criteria, and their assessment , 2008, Inf. Process. Manag..

[2]  Enrique Herrera-Viedma,et al.  A Selection Process Based on Additive Consistency to Deal with Incomplete Fuzzy Linguistic Information , 2010, J. Univers. Comput. Sci..

[3]  Michalis Sfakakis,et al.  User Behavior Tendencies on Data Collections in a Digital Library , 2002, ECDL.

[4]  Ying Zhang,et al.  Developing a holistic model for digital library evaluation , 2010, J. Assoc. Inf. Sci. Technol..

[5]  Francisco Herrera,et al.  Aggregation operators for linguistic weighted information , 1997, IEEE Trans. Syst. Man Cybern. Part A.

[6]  Enrique Herrera-Viedma,et al.  A Model Based on Fuzzy Linguistic Information to Evaluate the Quality of Digital Libraries , 2010, Int. J. Inf. Technol. Decis. Mak..

[7]  F. Clegg Simple Statistics: A Course Book for the Social Sciences , 1984 .

[8]  Enrique Herrera-Viedma,et al.  A new application of a fuzzy linguistic quality evaluation system in digital libraries , 2010, 2010 10th International Conference on Intelligent Systems Design and Applications.

[9]  László Kovács,et al.  Evaluation of Digital Libraries: Testbeds, Measurements, and Metrics , 2002 .

[10]  Edward A. Fox,et al.  Streams, structures, spaces, scenarios, societies (5s): A formal model for digital libraries , 2004, TOIS.

[11]  Nancy A. Vanhouse,et al.  Digital Library Use: Social Practice in Design and Evaluation , 2003 .

[12]  Sheng-Hshiung Tsaur,et al.  The evaluation of airline service quality by fuzzy MCDM. , 2002 .

[13]  Sally Jo Cunningham,et al.  A transaction log analysis of a digital library , 2000, International Journal on Digital Libraries.

[14]  Judy H. Jeng What is usability in the context of the digital library and how can it be measured , 2005 .

[15]  Norman W. H. Blaikie,et al.  Analyzing Quantitative Data: From Description to Explanation , 2000 .

[16]  Brian Shackel,et al.  Human factors for informatics usability , 1991 .

[17]  M. HamidR.Jamali,et al.  Site navigation and its impact on the content viewed by the virtual scholar: a deep log analysis , 2007, J. Inf. Sci..

[18]  Norman Blaikie,et al.  Analyzing Quantitative Data , 2012 .

[19]  Ann Blandford Understanding users' experiences: evaluation of digital libraries , 2004 .

[20]  Hao-Ren Ke,et al.  Exploring behavior of E-journal users in science and technology: Transaction log analysis of Elsevier's ScienceDirect OnSite in Taiwan , 2002 .

[21]  Heiko Schuldt,et al.  The Delos digital library reference model : foundations for digital libraries , 2007 .

[22]  R. Likert “Technique for the Measurement of Attitudes, A” , 2022, The SAGE Encyclopedia of Research Design.

[23]  Christine L. Borgman,et al.  Designing digital libraries for usability , 2003 .

[24]  Alireza Isfandyari-Moghaddam,et al.  Digital libraries in the mirror of the literature: issues and considerations , 2008, Electron. Libr..

[25]  Giannis Tsakonas,et al.  An ontological representation of the digital library evaluation domain , 2011, J. Assoc. Inf. Sci. Technol..

[26]  Ali Shiri,et al.  Metadata-enhanced visual interfaces to digital libraries , 2008, J. Inf. Sci..

[27]  Judy Jeng Evaluation of the New Jersey Digital Highway , 2008 .

[28]  G. Tsakonas,et al.  Evaluation of User Interaction in Digital Libraries , 2004 .

[29]  Chia-Yen Lin,et al.  Predicting information-seeking intention in academic digital libraries , 2009, Electron. Libr..

[30]  Brian Shackel,et al.  Usability - Context, framework, definition, design and evaluation , 1991, Interact. Comput..

[31]  Edward A. Fox,et al.  Automatic evaluation of digital libraries with 5SQual , 2009, J. Informetrics.

[32]  Xiaohua Hu,et al.  User tags versus expert-assigned subject terms: A comparison of LibraryThing tags and Library of Congress Subject Headings , 2010, J. Inf. Sci..

[33]  Tefko Saracevic,et al.  Evaluation of digital libraries: An overview , 2004 .

[34]  Jason A. Clark A usability study of the Belgian-American Research Collection: measuring the functionality of a digital library , 2004, OCLC Syst. Serv..

[35]  Ann Blandford,et al.  Analytical usability evaluation for digital libraries: a case study , 2004, JCDL.

[36]  Enrique Herrera-Viedma,et al.  A Consensus Model for Group Decision Making Problems with Unbalanced Fuzzy Linguistic Information , 2009, Int. J. Inf. Technol. Decis. Mak..

[37]  Marti A. Hearst,et al.  The state of the art in automating usability evaluation of user interfaces , 2001, CSUR.

[38]  Yong-Mi Kim,et al.  Gender role and the use of university library website resources: A social cognitive theory perspective , 2010, J. Inf. Sci..

[39]  N. Kano,et al.  Attractive Quality and Must-Be Quality , 1984 .

[40]  Cecilia Garibay,et al.  Evaluation of a Digital Library by Means of Quality Function Deployment (QFD) and the Kano Model , 2010 .

[41]  Xiaohui Liu,et al.  Evaluation of a personalized digital library based on cognitive styles: Adaptivity vs. adaptability , 2009, Int. J. Inf. Manag..

[42]  George Buchanan,et al.  The PRET A Rapporter framework: Evaluating digital libraries from the perspective of information work , 2008, Inf. Process. Manag..

[43]  Barbara Kitchenham,et al.  Procedures for Performing Systematic Reviews , 2004 .

[44]  Giannis Tsakonas,et al.  Evaluation of digital libraries : an insight to useful applications and methods , 2009 .

[45]  Gary Marchionini,et al.  Evaluating Digital Libraries: A Longitudinal and Multifaceted View , 2000, Libr. Trends.

[46]  Carol Peters,et al.  Evaluation of digital libraries , 2007, International Journal on Digital Libraries.

[47]  Patty Kostkova,et al.  User-Centered Evaluation Model for Medical Digital Libraries , 2009, K4HelP.

[48]  Hong Xie,et al.  Evaluation of digital libraries: Criteria and problems from users' perspectives , 2006 .

[49]  Judy Jeng What should we take into consideration when we talk about usability , 2009 .

[50]  Francisco Herrera,et al.  Group decision making with incomplete fuzzy linguistic preference relations , 2009, Int. J. Intell. Syst..

[51]  Cemal Atakan,et al.  An evaluation of the second survey on electronic databases usage at Ankara University Digital Library , 2008, Electron. Libr..

[52]  George Ghinea,et al.  Digital libraries: what do users want? , 2006, Online Inf. Rev..

[53]  Gary Marchionini,et al.  The People in Digital Libraries: Multifaceted Approaches to Assessing Needs and Impact , 1999 .

[54]  Enrique Herrera-Viedma,et al.  QUALITY EVALUATION OF DIGITAL LIBRARIES BASED ON LINGUISTIC INFORMATION , 2009 .

[55]  S. Jamieson Likert scales: how to (ab)use them , 2004, Medical education.

[56]  Giannis Tsakonas,et al.  Exploring usefulness and usability in the evaluation of open access digital libraries , 2008, Inf. Process. Manag..

[57]  Paul T. Jaeger,et al.  Developing multi-method, iterative, and user-centered evaluation strategies for digital libraries: Functionality, usability, and accessibility , 2006, ASIST.

[58]  Peter Johan Lor,et al.  To access is not to know: A critical reflection on A2K and the role of libraries with special reference to sub-Saharan Africa , 2010, J. Inf. Sci..

[59]  Richard T. Watson,et al.  Analyzing the Past to Prepare for the Future: Writing a Literature Review , 2002, MIS Q..

[60]  Tefko Saracevic,et al.  Digital Library Evaluation: Toward Evolution of Concepts , 2000, Libr. Trends.

[61]  Kar Yan Tam,et al.  Understanding User Acceptance of Digital Libraries: What are the Roles of Interface Characteristics, Organizational Context, and Individual Differences? , 2002, Int. J. Hum. Comput. Stud..

[62]  Álvaro Quijano-Solís,et al.  Evaluating a monolingual multinational digital library by using usability: An exploratory approach from a developing country , 2005 .

[63]  Pan Jun Kim,et al.  Domain analysis with text mining: Analysis of digital library research trends using profiling methods , 2010, J. Inf. Sci..

[64]  F. J. Cabrerizo A New Quality Evaluation Model Generating Recommendations to Improve the Digital Services Provided by the Academic Digital Libraries , 2011 .

[65]  Enrique Herrera-Viedma,et al.  Managing the consensus in group decision making in an unbalanced fuzzy linguistic context with incomplete information , 2010, Knowl. Based Syst..

[66]  Javier López Gijón,et al.  A fuzzy linguistic quality evaluation model for digital libraries , 2010, 2010 IEEE International Conference on Intelligent Systems and Knowledge Engineering.

[67]  David Bawden,et al.  Information (and library) science at City University London; 50 years of educational development , 2010, J. Inf. Sci..

[68]  Alan F. Smeaton,et al.  Personalisation and recommender systems in digital libraries , 2005, International Journal on Digital Libraries.

[69]  Charles Oppenheim,et al.  Managing and evaluating digital repositories , 2008, Inf. Res..

[70]  Francisco Herrera,et al.  Direct approach processes in group decision making using linguistic OWA operators , 1996, Fuzzy Sets Syst..

[71]  Norbert Fuhr,et al.  Digital Libraries: A Generic Classification and Evaluation Scheme , 2001, ECDL.

[72]  Alistair G. Sutcliffe,et al.  Evaluating the effectiveness of visual user interfaces for information retrieval , 2000, Int. J. Hum. Comput. Stud..

[73]  Judy Jeng,et al.  Usability Assessment of Academic Digital Libraries: Effectiveness, Efficiency, Satisfaction, and Learnability , 2005 .

[74]  Edward A. Fox,et al.  5SL: a language for declarative specification and generation of digital libraries , 2002, JCDL '02.

[75]  Edward A. Fox,et al.  Usability study of digital libraries: ACM, IEEE-CS, NCSTRL, NDLTD , 1999, International Journal on Digital Libraries.

[76]  Oscar Mauricio Serrano Jaimes,et al.  EVALUACION DE LA USABILIDAD EN SITIOS WEB, BASADA EN EL ESTANDAR ISO 9241-11 (International Standard (1998) Ergonomic requirements For office work with visual display terminals (VDTs)-Parts II: Guidance on usability , 2012 .

[77]  Maria Pinto,et al.  Insights into translation students’ information literacy using the IL-HUMASS survey , 2010, J. Inf. Sci..