Evaluation of hypermedia educational systems: criteria and imperfect measures

Educational systems have to be assessed to determine whether their user interface, didactic method, instructional materials and interaction mechanisms help users reach their learning and teaching goals. With this purpose, educational developers need to rely on a set of clear, concrete and measurable quality criteria. In this paper, the authors propose an evaluation framework made up of a number of criteria to test the utility and usability of educational hypermedia systems. They also discuss that measuring these criteria entails different forms of imprecision and uncertainty, that call for the user of representation formalisms like fuzzy set theory. The framework presented here is aimed not only at helping educational developers to evaluate their systems, but also to provide them with some guidance to address educational requirements during the design process.

[1]  Ben Shneiderman,et al.  Structural analysis of hypertexts: identifying hierarchies and useful metrics , 1992, TOIS.

[2]  Simone Santini,et al.  Similarity Measures , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Jennifer Preece,et al.  Lurker demographics: counting the silent , 2000, CHI.

[4]  Jenny Preece,et al.  A Guide to Usability: Human Factors in Computing , 1993 .

[5]  Robert Pryor,et al.  An Application of a Computerized Fuzzy Graphic Rating Scale to the Psychological Measurement of Individual Differences , 1988, Int. J. Man Mach. Stud..

[6]  Paloma Díaz,et al.  Evaluation Criteria for Hypermedia Educational Systems , 2001, Computers and Education. Towards an Interconnected Society.

[7]  George J. Klir,et al.  Uncertainty-Based Information , 1999 .

[8]  Dimitris Christodoulakis,et al.  Measuring the readability and maintainability of hyperdocuments , 1995, J. Softw. Maintenance Res. Pract..

[9]  Gregg C. Vanderheiden,et al.  Web content accessibility guidelines 1.0 , 2001, INTR.

[10]  Tracy Irani,et al.  If We Build It, Will They Come? The Effects of Experience and Attitude on Traditional-Aged Students' Views of Distance Education , 2000 .

[11]  Lotfi A. Zadeh,et al.  Toward a theory of fuzzy information granulation and its centrality in human reasoning and fuzzy logic , 1997, Fuzzy Sets Syst..

[12]  P. Smets.  Imperfect information : Imprecision-Uncertainty , 1999 .

[13]  Franca Garzotto,et al.  Hypermedia design, analysis, and evaluation issues , 1995, CACM.

[14]  Jung-Kook Hong,et al.  Development and evaluation of hypermedia for museum education: validation of metrics , 1995, TCHI.

[15]  Paloma Díaz,et al.  The evaluation of a hypermedia learning environment: the CESAR experience , 1996 .

[16]  Nick Hammond,et al.  Learning support environments: Rationale and evaluation , 1990 .

[17]  Arthur C. Graesser,et al.  Assessing the Pedagogical Effectiveness and Conversational Appropriateness in Three Versions of AutoTutor , 2001 .

[18]  Jakob Nielsen,et al.  The Anti-Mac interface , 1996, CACM.

[19]  Philippe Smets,et al.  Imperfect Information: Imprecision and Uncertainty , 1996, Uncertainty Management in Information Systems.

[20]  Melody Y. Ivory.  Web TANGO: towards automated comparison of information-centric web site designs , 2000, CHI Extended Abstracts.

[21]  Miguel A. Sicilia,et al.  Computer-Aided Usability Evaluation : A Questionnaire Case Study , 2001 .

[22]  Rachel Harrison,et al.  Applying Metrics to the Evaluation of Educational Hypermedia Applications , 1998, J. Univers. Comput. Sci..

[23]  Camino Fernández,et al.  Assessing the utility of an interactive electronic book for learning the Pascal programming language , 2000, IEEE Trans. Educ..

[24]  José Bravo,et al.  Computers and Education. Towards an Interconnected Society , 2001 .

[25]  Gary Perlman.  Practical usability evaluation , 1995, CHI 95 Conference Companion.

[26]  Michael Smithson,et al.  Fuzzy Set Theory and Applications in Psychology , 1999 .

[27]  Didier Dubois,et al.  Fuzzy information engineering: a guided tour of applications , 1997 .