Scenario-based Group Usability Testing as a Mixed Methods Approach to the Evaluation of Three-Dimensional Virtual Learning Environments

Although virtual reality became popular technology whose application is recognized in various domains, the field generally still lacks a widespread culture of usability. This is also evident when considering environments intended for learning, specifically virtual learning environments (VLEs). According to our findings, it is clear that there is a growing need for systematic evaluation approach to help with the design and development of usable learner-centered VLE solutions. After comprehensive introductory background and state of the art in the field, this article provides an insight into Scenario-based Group Usability Testing (ScerGUT), a mixed methods approach to the evaluation of three-dimensional VLEs which integrates several different methods of usability testing with measurements of educational value. While the majority of the existing work has made use of usually one single usability assessment technique, ScerGUT employs a number of methods putting in focus users and user testing. To examine efficiency and applicability of the approach, empirical validation is conducted as a case study of particular VLE. The contribution of the article is twofold: (a) ScerGUT as a mixed methods approach to the evaluation of VLEs, which brings new scientific value and could help other researchers and (b) ScerGUT’s application to a particular VLE, which brings quantitative and qualitative results, thus providing an insight into ease of use and educational value of specific VLE.

[1]  Tim Marsh,et al.  Co-operative Evaluation of a Desktop Virtual Reality System , 1999 .

[2]  Zahira Merchant,et al.  The learner characteristics, features of desktop 3D virtual reality environments, and college chemistry instruction: A structural equation modeling analysis , 2012, Comput. Educ..

[3]  Panagiotis Zaharias,et al.  Minimizing Attrition in Online Degree Courses. , 2009 .

[4]  Chin-Wen Liao,et al.  A Virtual Experiential Learning and Students' Ill-Structured Problem-Solving Ability , 2014, Interact. Comput..

[5]  Spyros Vosinakis,et al.  Interaction design studio learning in virtual worlds , 2013, Virtual Reality.

[6]  Vicky Maratou,et al.  Information Security Awareness through a Virtual World: An end-user requirements analysis , 2015 .

[7]  Franca Garzotto,et al.  Quality and Potential for Adoption of Usability Evaluation Methods , 2008, J. Web Eng..

[8]  Cristina Botella,et al.  Acceptability and Satisfaction of an ICT-based Training for University Teachers , 2015, J. Educ. Technol. Soc..

[9]  Ergonomic requirements for office work with visual display terminals ( VDTs ) — Part 11 : Guidance on usability , 1998 .

[10]  Aimee deNoyelles,et al.  Exploring the Dimensions of Self-Efficacy in Virtual World Learning: Environment, Task, and Content , 2014 .

[11]  Andrina Granic,et al.  Experience with usability evaluation of e-learning systems , 2008, Universal Access in the Information Society.

[12]  Deborah J. Mayhew,et al.  The usability engineering lifecycle , 1998, CHI Conference Summary.

[13]  Doug A. Bowman,et al.  A Survey of Usability Evaluation in Virtual Environments: Classification and Comparison of Methods , 2002, Presence: Teleoperators & Virtual Environments.

[14]  Andrina Granic,et al.  Usability Testing and Expert Inspections Complemented by Educational Evaluation: A Case Study of an e-Learning Platform , 2011, J. Educ. Technol. Soc..

[15]  Andrina Grani,et al.  Personalized Context-Aware Recommendations In 3 D Virtual Learning Environments , 2015 .

[16]  Mansooreh Mollaghasemi,et al.  Usability engineering of virtual environments (VEs): identifying multiple criteria that drive effective VE system design , 2003, Int. J. Hum. Comput. Stud..

[17]  Gavin McArdle,et al.  Virtual reality for collaborative e-learning , 2008, Comput. Educ..

[18]  S. Vosinakis,et al.  A Framework for Problem-Based Learning Activities in Virtual Worlds , 2011 .

[19]  Mark Stansfield,et al.  UNITE: Enhancing Students' Self-efficacy through the Use of a 3D Virtual World , 2015, J. Univers. Comput. Sci..

[20]  Maria Roussou,et al.  Design and Evaluation of a Real-World Virtual Environment for Architecture and Urban Planning , 2007, PRESENCE: Teleoperators and Virtual Environments.

[21]  Luca Chittaro,et al.  Web3D technologies in learning, education and training: Motivations, issues, opportunities , 2007, Comput. Educ..

[22]  Angeliki Poulymenakou,et al.  Developing a Usability Evaluation Method for e-Learning Applications: Beyond Functional Usability , 2009, Int. J. Hum. Comput. Interact..

[23]  Daizha Norris.  Naval Air Warfare Center , 2018 .

[24]  José Ramón Calvo-Ferrer,et al.  Educational games as stand-alone learning tools and their motivational effect on L2 vocabulary acquisition and perceived learning gains , 2017, Br. J. Educ. Technol..

[25]  Panagiotis Zaharias,et al.  Usability in the Context of e-Learning: A Framework Augmenting 'Traditional' Usability Constructs with Instructional Design and Motivation to Learn , 2009, Int. J. Technol. Hum. Interact..

[26]  B. L. William Wong,et al.  Usability evaluation of non-immersive, desktop, photo-realistic virtual environments , 2004 .

[27]  J. B. Brooke,et al.  SUS: a retrospective , 2013 .

[28]  John R. Wilson,et al.  Systematic Usability Evaluation and Design Issues for Collaborative Virtual Environments , 2003, Presence: Teleoperators & Virtual Environments.

[29]  Olga De Troyer,et al.  Authoring Adaptive 3D Virtual Learning Environments , 2014, Int. J. Virtual Pers. Learn. Environ..

[30]  Leslie Jarmon,et al.  Virtual world teaching, experiential learning, and assessment: An interdisciplinary communication course in Second Life , 2009, Comput. Educ..

[31]  William Buxton,et al.  Usability evaluation considered harmful (some of the time) , 2008, CHI.

[32]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[33]  Antonella De Angeli,et al.  eLSE Methodology: a Systematic Approach to the e-Learning Systems Evaluation , 2006, J. Educ. Technol. Soc..

[34]  Asmaa Alsumait,et al.  Usability heuristics evaluation for child e-learning applications , 2009, J. Softw..

[35]  David C. Schwebel,et al.  Usability and feasibility of an internet-based virtual pedestrian environment to teach children to cross streets safely , 2013, Virtual Reality.

[36]  Kasper Hornbæk,et al.  Tracing impact in a usability improvement process , 2008, Interact. Comput..

[37]  Alison F. Doubleday,et al.  The virtual anatomy laboratory: Usability testing to improve an online learning resource for anatomy education , 2011, Anatomical sciences education.

[38]  Junpeng Zhang,et al.  A Peek at the Position of Pedagogical Aspects in Usability Evaluation of E-learning System - A Literature Review of Usability Evaluation of E-learning System conducted since 2000 , 2010 .

[39]  Turkan Karakus,et al.  The Development of a Criteria List for the Selection of 3D Virtual Worlds to Design an Educational Environment / Razrada popisa kriterija za odabir 3D virtualnih svjetova i izradu obrazovnog okruženja , 2015 .

[40]  Teresa Roselli,et al.  An approach to usability evaluation of e-learning applications , 2006, Universal Access in the Information Society.

[41]  Thrasyvoulos Tsiatsos,et al.  Fostering collaborative learning in Second Life: Metaphors and affordances , 2010, Comput. Educ..

[42]  Maria Virvou,et al.  On the usability and likeability of virtual reality games for education: The case of VR-ENGAGE , 2008, Comput. Educ..

[43]  Dominique L. Scapin,et al.  Please Scroll down for Article International Journal of Human-computer Interaction Comparing Inspections and User Testing for the Evaluation of Virtual Environments Comparing Inspections and User Testing for the Evaluation of Virtual Environments , 2022 .

[44]  Nikolaos M. Avouris,et al.  The effective combination of hybrid usability methods in evaluating educational applications of ICT: Issues and challenges , 2008, Education and Information Technologies.

[45]  Jennifer Preece,et al.  Predicting quality in educational software: Evaluating For Learning, Usability and the Synergy between Them , 1999, Interact. Comput..

[46]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[47]  Samuel Ssemugabi,et al.  Effectiveness of heuristic evaluation in usability evaluation of e-learning applications in higher education , 2010, South Afr. Comput. J..

[48]  Chee Siong Teh,et al.  Interface Design Usability Testing of Virtual Reality (VR)-Based Learning Environment , 2009 .

[49]  Alistair G. Sutcliffe,et al.  Evaluating the usability of virtual reality user interfaces , 2000, Behav. Inf. Technol..

[50]  Chwen Jen Chen,et al.  A feasible group testing framework for producing usable virtual reality learning applications , 2015, Virtual Reality.

[51]  S. A. Becker,et al.  NMC Horizon Report: 2016 Higher Education Edition , 2015 .