A Framework for Evaluating the Quality of Multimedia Learning Resources

This article presents the structure and theoretical foundations of the Learning Object Review Instrument (LORI), an evaluation aid available through the E-Learning Research and Assessment Network at http://www.elera.net. A primary goal of LORI is to balance assessment validity with efficiency of the evaluation process. The instrument enables learning object users to create reviews consisting of ratings and comments on nine dimensions of quality: content quality, learning goal alignment, feedback and adaptation, motivation, presentation design, interaction usability, accessibility, reusability, and standards compliance. The article presents research and practices relevant to these dimensions and describes how each dimension can be interpreted to evaluate multimedia learning resources.

[1]  Theo Mandel,et al.  Quality technical information: paving the way for usable print and web interface design , 2002, AJCD.

[2]  David A. Sewry,et al.  Describing online learning content to facilitate resource discovery and sharing: the development of the RU LOM Core , 2006, J. Comput. Assist. Learn..

[3]  J. Sweller,et al.  Cognitive Load Theory and Complex Learning: Recent Developments and Future Directions , 2005 .

[4]  Chris J. H. Fowler,et al.  Learning technology and usability: a framework for understanding courseware , 1999, Interact. Comput..

[5]  L. L Lohr,et al.  Designing the instructional interface , 2000 .

[6]  Eileen Scanlon,et al.  Contexts for evaluating educational software , 1999, Interact. Comput..

[7]  Lauren B. Resnick,et al.  Benchmarking and Alignment of Standards and Testing , 2004 .

[8]  John C. Nesbit,et al.  Learning Object Evaluation and Convergent Participation: Tools for Professional Development in E-Learning , 2004, Computers and Advanced Technology in Education.

[9]  Michael J. Sanger,et al.  An Analysis of College Chemistry Textbooks As Sources of Misconceptions and Errors in Electrochemistry , 1999 .

[10]  V. Aleven,et al.  Help Seeking and Help Design in Interactive Learning Environments , 2003 .

[11]  R. Mayer,et al.  Nine Ways to Reduce Cognitive Load in Multimedia Learning , 2003 .

[12]  Gloria Dall'Alba,et al.  Textbook treatments and students' understanding of acceleration , 1993 .

[13]  Olusola O. Adesope,et al.  Learning With Concept and Knowledge Maps: A Meta-Analysis , 2006 .

[14]  Diane Cordry Golden.  Instructional Software Accessibility: A Status Report , 2002 .

[15]  Jaan Mikk.  Experimental Evaluation of Textbooks and Multimedia. , 2002 .

[16]  Mark Guzdial,et al.  Software-Realized Scaffolding to Facilitate Programming for Science Learning , 1994, Interact. Learn. Environ..

[17]  Michael J. Hannafin,et al.  Teaching and learning in digital environments: The resurgence of resource-based learning , 2001 .

[18]  Jakob Nielsen,et al.  Enhancing the explanatory power of usability heuristics , 1994, CHI '94.

[19]  Allyson F. Hadwin,et al.  CoNoteS2: A Software Tool for Promoting Self-Regulation , 2001 .

[20]  Gregg C. Vanderheiden,et al.  Web content accessibility guidelines 1.0 , 2001, INTR.

[21]  De Laurentiis.  How To Recognize Excellent Educational Software. , 1993 .

[22]  Atsusi Hirumi,et al.  In Search of Quality: An Analysis of e-Learning Guidelines and Specifications. , 2005 .

[23]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[24]  Stephen K. Reed,et al.  Cognitive Architectures for Multimedia Learning , 2006 .

[25]  Chen-Lin C. Kulik,et al.  The Instructional Effect of Feedback in Test-Like Events , 1991 .

[26]  Gary R. Morrison,et al.  The Effects of Feedback and Incentives on Achievement in Computer-Based Instruction , 1995 .

[27]  Chi-Yan Tsui,et al.  Motivational Aspects of Learning Genetics with Interactive Multimedia , 2004 .

[28]  B. Bloom.  The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring , 1984 .

[29]  Robert D. Tennyson,et al.  MAIS: An intelligent learning system , 1988 .

[30]  S. A. Cohen,et al.  Instructional Alignment: Searching for a Magic Bullet , 1987 .

[31]  Alberta Labour.  Section 508 Standards , 2007 .

[32]  Sebastiano Bagnara,et al.  Multimedia systems in distance education: effects of usability on learning , 1999, Interact. Comput..

[33]  Greg Kearsley.  Online help systems: design and implementation , 1988 .

[34]  Joseph Krajcik,et al.  The design of guided learner-adaptable scaffolding in interactive learning environments , 1996, CHI.

[35]  G. Hanley,et al.  MERLOT: A faculty-focused Web site of educational resources , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[36]  Donald F. Dansereau,et al.  Multirelational semantic maps , 1989 .

[37]  James E. Driskell,et al.  Games, Motivation, and Learning: A Research and Practice Model , 2002 .

[38]  Deborah J. Mayhew,et al.  Principles and Guidelines in Software User Interface Design , 1991 .

[39]  Erik Duval,et al.  Standardized uniqueness: oxymoron or vision of the future? [IEEE standards] , 2006, Computer.

[40]  John C. Nesbit,et al.  Teachers Need Simple, Effective Tools to Evaluate Learning Objects: Enter eLera.net , 2004, CATE.

[41]  Allison Littlejohn,et al.  Reusing Online Resources: A Sustainable Approach to E-Learning , 2003 .

[42]  John Keller,et al.  Learner motivation and E-learning design: A multinationally validated process , 2004 .

[43]  J. Eccles,et al.  Expectancy-Value Theory of Achievement Motivation. , 2000, Contemporary educational psychology.

[44]  Ok-Choon Park,et al.  Adaptive Instructional Systems , 2007 .

[45]  Susan E. Metros.  Visualizing knowledge in new educational environments: a course on learning objects , 2005 .

[46]  E. Deci,et al.  Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. , 2000, Contemporary educational psychology.

[47]  Norman L. Webb,et al.  Alignment of an Alternate Assessment with State Academic Standards , 2005 .

[48]  Robert A. Reiser,et al.  The Effects of Self-Regulation and Instructional Control on Performance and Motivation in Computer-based Instruction , 2000 .

[49]  John Hedberg,et al.  Designing Multimedia: Seven Discourses , 2004 .

[50]  Jeremy Roschelle,et al.  ESCOT: Coordinating the Influence of R&D and Classroom Practice to Produce Educational Software From Reusable Components , 2004, Interact. Learn. Environ..

[51]  David L. Ryan-Jones,et al.  Designing Instruction with Learning Objects. , 2002 .

[52]  Gregory Schraw,et al.  Increasing Situational Interest in the Classroom , 2001 .

[53]  Phil Parette,et al.  Millennial Students with Mild Disabilities and Emerging Assistive Technology Trends , 2005 .

[54]  Mubina Hassanali Kirmani,et al.  Diversity and Technology: Classroom Implications of the Digital Divide , 2005 .

[55]  Rose Luckin,et al.  Ecolab: the development and evaluation of a Vygotskian design framework , 1999 .

[56]  Christopher D. Wickens,et al.  An introduction to human factors engineering , 1997 .

[57]  Malcolm L. Fleming,et al.  Instructional Message Design: Principles from the Behavioral and Cognitive Sciences , 1993 .

[58]  Myke Gluck,et al.  Visual Explanations: Images and Quantities, Evidence and Narrative , 1997, Inf. Process. Manag..

[59]  John C. Nesbit,et al.  A Convergent Participation Model for Evaluation of Learning Objects , 2002 .

[60]  Tony Koppi,et al.  Learning objects, repositories, sharing and reusability , 2005 .

[61]  Theo J. Bastiaens,et al.  The impact of intrinsic motivation on e-learning in authentic computer tasks , 2004, J. Comput. Assist. Learn..

[62]  Andrew C. Porter,et al.  Measuring the Content of Instruction: Uses in Research and Practice , 2002 .

[63]  Jennifer Preece,et al.  A psychological investigation of long retrieval times on the World Wide Web , 1998, Interact. Comput..

[64]  Patrick E. Parrish,et al.  The trouble with learning objects , 2004 .

[65]  Siobhan Chapman.  Logic and Conversation , 2005 .

[66]  John C. Nesbit,et al.  The Teaching of Quality: Convergent Participation for the Professional Development of Learning Object Designers , 2004 .

[67]  Jerry Zhigang Li,et al.  Evaluating Learning Objects Across Boundaries: The Semantics of Localization , 2006, Int. J. Distance Educ. Technol..

[68]  Jean McKendree,et al.  Effective Feedback Content for Tutoring Complex Skills , 1990, Hum. Comput. Interact..

[69]  Michael D. Bush.  Connecting Instructional Design to International Standards for Content Reusability. , 2002 .

[70]  Nikolaos Avouris,et al.  Evaluation of Distance-Learning Environments: Impact of Usability on Student Performance , 2001 .

[71]  John C. Nesbit,et al.  Learning Object Evaluation: Computer-Mediated Collaboration And Inter-Rater Reliability , 2003 .

[72]  Richard H. Hall,et al.  The Effects of Graphical Postorganization Strategies on Learning From Knowledge Maps , 1999 .

[73]  Ira P. Goldstein,et al.  The genetic graph: a representation for the evolution of procedural knowledge , 1979 .

[74]  van SchaikPaul,et al.  The effect of spatial layout of and link colour in web pages on performance in a visual search task and an interactive search task , 2003 .

[75]  Paul van Schaik,et al.  The effect of spatial layout of and link colour in web pages on performance in a visual search task and an interactive search task , 2003, Int. J. Hum. Comput. Stud..

[76]  Diana Laurillard,et al.  Affordances for Learning in a Non-linear Narrative Medium , 2000 .

[77]  David H. Jonassen,et al.  Handbook of Research for educational Communications and Technology , 1997 .

[78]  Barbara S. Chaparro,et al.  The world wide wait: effects of delays on user performance , 2000 .

[79]  Donna M. Gollnick,et al.  Multicultural Education for Exceptional Children. ERIC Digest #E498. , 1991 .

[80]  R. Harden.  A new vision for distance learning and continuing medical education , 2005, The Journal of continuing education in the health professions.

[81]  J. Keller.  Strategies for stimulating the motivation to learn , 1987 .

[82]  Rory McGreal.  Online Education Using Learning Objects , 2004 .

[83]  D. Deregt,et al.  Introduction and History. , 2008, Bulletin of the Medical Library Association.

[84]  Ioanna Vekiri.  What Is the Value of Graphical Displays in Learning? , 2002 .

[85]  Marilyn McClelland.  Metadata Standards for Educational Resources , 2003, Computer.

[86]  Jennifer Preece,et al.  Predicting quality in educational software: Evaluating For Learning, Usability and the Synergy between Them , 1999, Interact. Comput..

[87]  Jason Palmeri,et al.  Disability Studies, Cultural Analysis, and the Critical Practice of Technical Communication Pedagogy , 2006 .

[88]  Cheryl Rappaport Liebling.  Achieving Standards-Based Curriculum Alignment through Mindful Teaching. , 1997 .

[89]  Marcia A. Mardis,et al.  Leading the Wave of Science and Mathematics Learning Innovation: NSDL Resources. , 2003 .

[90]  Margaret Haughey,et al.  Evaluating Learning Objects for Schools , 2005 .

[91]  Michael G. Paciello.  Web Accessibility for People with Disabilities , 2000 .

[92]  Alfred Bork,et al.  Multimedia in Learning , 2001 .

[93]  John R. Anderson,et al.  Cognitive Tutors: Lessons Learned , 1995 .

[94]  Larisa V. Shavinina,et al.  Psychological Evaluation of Educational Multimedia Applications , 1999 .

[95]  Brent G. Wilson,et al.  Affordances and Constraints of the Internet for Learning and Instruction. , 1996 .