State-ofthe-Art-Matrix Analysis for Usability of Learning Management Systems

This is a research study to explore trends, gaps, and issues in the literature of the usability of Learning Management Systems (LMS). The authors utilized the State-of-the-Art Matrix analysis, which is a research method that has been used extensively in the last decade. It is a systematic evaluation of existing research by using several statistical methods. Pareto analysis and Histograms are part of this analysis. The analysis revealed several gaps: (1) engineering students have not been the main focus of research in any studies, (2) there is no research that compares usability of LMS between different academic disciplines, (3) there is no modeling effort for understanding if engineering students and instructors need different LMS design than other disciplines, (4) primary framework development for evaluating LMS has declined, (5) discount usability methods (heuristics) have been mostly preferred for the evaluation of LMS ignoring effectiveness and efficiency performance measures related to LMS usage, (6) there are very limited studies incorporating usability design with instructional and accessibility design, (7) there are very limited studies investigating LMS usability with regards to occupational training, (8) there are many researchers who mentioned the significance of research on usability of mobile e-learning platforms. The results of this study established a basis for future work and the authors will study LMS usability for engineering students and instructors by future empirical studies.

[1]  J. Gerring.  A case study , 2011, Technology and Society.

[2]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[3]  Laurie P. Dringus.  An Iterative Usability Evaluation Procedure for Interactive Online Courses , 1995 .

[4]  Marshall G. Jones,et al.  User Interface Design Guidelines for Computer-Based Learning Environments , 1999 .

[5]  Rosemary Hayes.  Exploring Discount Usability Methods to Assess the Suitability of Online Course Delivery Products , 1999 .

[6]  Peter Albion,et al.  Multimedia: , 2020, 'Komt pelgrims, komt hier'.

[7]  Susan Weinschenk,et al.  Usability Is Good Business , 1999 .

[8]  Allison Peraya,et al.  Creating web-integrated learning environments: An analysis of WebCT authoring tools in respect to usability , 2000 .

[9]  Nikolaos Avouris,et al.  Evaluation of Distance-Learning Environments: Impact of Usability on Student Performance , 2001 .

[10]  Mei Wang,et al.  Evaluating the usability of Web-based learning tools , 2002, J. Educ. Technol. Soc..

[11]  Thomas C. Reeves,et al.  Usability and Instructional Design Heuristics for E-Learning Evaluation , 2002 .

[12]  Brent Auernheimer,et al.  Guidelines for designing evaluations of Web-based instructional materials , 2002, Proceedings of the 35th Annual Hawaii International Conference on System Sciences.

[13]  Albert L. Ingram.  Usability of Alternative Web Course Structures , 2002 .

[14]  Angeliki Poulymenakou,et al.  Designing On-Line Learning Courses : Implications for Usability , 2002 .

[15]  Janet Wesson,et al.  Formal Usability Testing of Interactive Educational Software: A Case Study , 2002, Usability.

[16]  Peter Chalk,et al.  Evaluating the use of a virtual learning environment for teaching aspects of HCI , 2002, ITiCSE '02.

[17]  F. Colace,et al.  Evaluating on-line learning platforms: a case study , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[18]  K. Silius,et al.  A multidisciplinary tool for the evaluation of usability , pedagogical usability , accessibility and informational quality of web-based courses , 2003 .

[19]  Slavomir Stankov,et al.  Usability Evaluation Methodology for Web-based Educational Systems , 2004 .

[20]  Rose Mary Wentling,et al.  The Relationship between National Culture and the Usability of an E-learning System , 2004 .

[21]  Luca Botturi,et al.  MiLE : Systematic Usability Evaluation for E-learning Web Applications , 2004 .

[22]  Bee Bee Chua,et al.  Applying the ISO 9126 model to the evaluation of an e- learning system , 2004 .

[23]  D. Bremer,et al.  A Comparison of Two Learning management Systems : Moodle vs Blackboard , 2005 .

[24]  L.P. Dringus,et al.  An adaptable usability heuristic checklist for online courses , 2005, Proceedings Frontiers in Education 35th Annual Conference.

[25]  S. Retalis,et al.  Evaluating the usability of Web-based Learning Management Systems , 2005 .

[26]  Teresa Roselli,et al.  An approach to usability evaluation of e-learning applications , 2006, Universal Access in the Information Society.

[27]  Szu-Yuan Sun,et al.  Usability, quality, value and e-learning continuance decisions , 2005, Comput. Educ..

[28]  Panagiotis Zaharias,et al.  A usability evaluation method for e-learning: focus on motivation to learn , 2006, CHI Extended Abstracts.

[29]  Rosa Lanzilotti,et al.  SYSTEMATIC EVALUATION OF E-LEARNING SYSTEMS , 2006 .

[30]  Luca Botturi,et al.  Evaluating LMS Usability for Enhanced eLearning Experience , 2006 .

[31]  Antonella De Angeli,et al.  Systematic evaluation of e-learning systems: an experimental validation , 2006, NordiCHI '06.

[32]  William R. Watson,et al.  An Argument for Clarity: What are Learning Management Systems, What are They Not, and What Should They Become? , 2007 .

[33]  M. Machado,et al.  Blackboard vs. moodle: Comparing user experience of learning management systems , 2007, 2007 37th Annual Frontiers In Education Conference - Global Engineering: Knowledge Without Borders, Opportunities Without Passports.

[34]  Samuel Ssemugabi,et al.  A comparative study of two usability evaluation methods using a web-based e-learning application , 2007, SAICSIT '07.

[35]  Samuel Ssemugabi,et al.  Usability and Learning: A Framework for Evaluation of Web-Based e-Learning Applications , 2007 .

[36]  Tanja Arh,et al.  Evaluation and selection of the most applicable learning management system , 2007 .

[37]  Andrina Granic,et al.  Experience with usability evaluation of e-learning systems , 2008, Universal Access in the Information Society.

[38]  Claudio Kirner,et al.  Usability Evaluation Of The Moodle System From The Teachers' Perspective , 2008, e-Learning.

[39]  Sime Arsenovski,et al.  Evaluating usability in learning management system moodle , 2008, ITI 2008 - 30th International Conference on Information Technology Interfaces.

[40]  J. Boticario,et al.  Usability in e-Learning Platforms : heuristics comparison between Moodle , Sakai and dotLRN , 2008 .

[41]  Dusan Starcevic,et al.  Usability Issues of e-Learning Systems: Case-Study for Moodle Learning Management System , 2008, OTM Workshops.

[42]  Florence Martin.  Blackboard as the Learning Management System of a Computer Literacy Course , 2009 .

[43]  Sevgi Özkan,et al.  Multi-dimensional students' evaluation of e-learning systems in the higher education context: An empirical investigation , 2009, Comput. Educ..

[44]  Yu-chu Yeh,et al.  Development of design criteria and evaluation scale for web-based learning platforms , 2009 .

[45]  Angeliki Poulymenakou,et al.  Developing a Usability Evaluation Method for e-Learning Applications: Beyond Functional Usability , 2009, Int. J. Hum. Comput. Interact..

[46]  Adrian Besimi,et al.  Usability issues while building a new LMS , 2009, Proceedings of the ITI 2009 31st International Conference on Information Technology Interfaces.

[47]  Yuqing Guo,et al.  A case study of usability testing on an asynchronous e-Learning platform , 2009, 2009 Joint Conferences on Pervasive Computing (JCPC).

[48]  Asmaa Alsumait,et al.  Usability heuristics evaluation for child e-learning applications , 2009, J. Softw..

[49]  Alexander Blecken,et al.  Usability Evaluation of a Learning Management System , 2010, 2010 43rd Hawaii International Conference on System Sciences.

[50]  Khadijah Abdul Rahman,et al.  The Effectiveness of Learning Management System (LMS) Case Study at Open University Malaysia (OUM), Kota Bharu Campus , 2010 .

[51]  Asil Oztekin,et al.  UseLearn: A novel checklist and usability evaluation method for eLearning systems by criticality metric analysis , 2010 .

[52]  Erhard Zorn,et al.  Evaluation of a MOODLE Based Learning Management System Applied at Berlin Institute of Technology Based on ISO-9126 , 2010 .

[53]  Zafer Unal,et al.  Evaluating and Comparing the Usability of Web-Based Course Management Systems , 2011 .

[54]  Paula Kotzé,et al.  Course Management Systems from a Usability Perspective , 2011 .

[55]  Nor Azan Mat Zin,et al.  2011 International Conference on Electrical Engineering and Informatics 17-19 July 2011 , Bandung , Indonesia Heuristic Evaluation of Distance Learning Management System Interface , 2011 .

[56]  Mario G. Beruvides,et al.  The state-of-the-art matrix analysis for cost-justification of usability research , 2011 .

[57]  Michalis Xenos,et al.  USABILITY INSPECTION THROUGH HEURISTIC EVALUATION IN E- LEARNING ENVIRONMENTS: THE LAMS CASE , 2011 .

[58]  Andrina Granic,et al.  Usability Testing and Expert Inspections Complemented by Educational Evaluation: A Case Study of an e-Learning Platform , 2011, J. Educ. Technol. Soc..

[59]  Hoda Baytiyeh.  Work in progress — Open source usability evaluation: The case of Moodle , 2011, 2011 Frontiers in Education Conference (FIE).

[60]  Asil Oztekin,et al.  A decision support system for usability evaluation of web-based information systems , 2011, Expert Syst. Appl..

[61]  Michalis Nik Xenos,et al.  Perceived Usability Evaluation of Learning Management Systems: A First Step towards Standardization of the System Usability Scale in Greek , 2012, 2012 16th Panhellenic Conference on Informatics.

[62]  José Creissac Campos,et al.  A literature review about usability evaluation methods for e-learning platforms. , 2012, Work.

[63]  N. Lalande,et al.  Blackboard vs. Desire2Learn: A system administrator's perspective on usability , 2012, International Conference on Education and e-Learning Innovations.

[64]  Jerome Turner,et al.  The pivotal role of staff user experiences in Moodle and the potential impact on student learning , 2013, 2013 Second International Conference on E-Learning and E-Technologies in Education (ICEEE).

[65]  Selim Zaim,et al.  A machine learning-based usability evaluation method for eLearning systems , 2013, Decis. Support Syst..

[66]  Marko Hölbl,et al.  Usability and Privacy Aspects of Moodle: Students' and Teachers' Perspective , 2013, Informatica.

[67]  Tengku Siti Meriam Tengku Wook,et al.  User Testing for Moodle Application , 2013 .

[68]  Muhammad Shabbir Abbasi,et al.  Usability Testing of an E- Learning System: A Comparative study of two Evaluation Techniques , 2014 .

[69]  Selvarajah Thuseethan,et al.  Usability Evaluation of Learning Management Systems in Sri Lankan Universities , 2014, ArXiv.

[70]  Walter Franklin,et al.  Heuristic Evaluation of Usability - a Case study with the Learning Management Systems (LMS) of IFPE , 2014 .

[71]  Zafer Unal,et al.  Investigating and Comparing User Experiences of Course Management Systems: BlackBoard vs. Moodle , 2014 .

[72]  Nicholas Kipkurui Kiget,et al.  Evaluating Usability of E-Learning Systems in Universities , 2014 .

[73]  Mario G. Beruvides,et al.  A review of the state–of–the–art research on white–collar/knowledge–worker productivity , 2014 .

[74]  Priyavahani Subramanian,et al.  A Study of Comparison between Moodle and Blackboard based on Case Studies for Better LMS , 2014 .

[75]  Marmelo V. Abante,et al.  An Evaluation of Integrated Dynamic Learning Environment (IDLE) at College of Saint John Paul II Arts and Sciences on ISO-9126 , 2015 .

[76]  Mario G. Beruvides,et al.  High Reliability Organization Research: A Literature Review for Health Care , 2015 .

[77]  Joel S. Mtebe,et al.  Heuristics for evaluating usability of Learning Management Systems in Africa , 2015, 2015 IST-Africa Conference.

[78]  Murni Mahmud,et al.  Comparing the Usability and Users Acceptance of Open Sources Learning Management System (LMS) , 2015 .