Evaluating the usability and instructional design quality of Interactive Virtual Training for Teachers (IVT-T)

Simulation offers unique affordances over traditional training (e.g., remote access, mastery learning experiences, immediate feedback) relevant to teacher training in behavior management. This study describes a user-based evaluation of Interactive Virtual Training for Teachers (IVT-T). The study involved observing representative users (seven advanced education majors) perform benchmark tasks with the system, complete rating scales, and participate in interviews to evaluate the usability and instructional design quality of IVT-T. Global usability ratings based on established usability rating scales suggested IVT-T was adequately usable while observations of user performance and semi-structured interviews revealed design shortcomings that impeded effective user performance and informed ways to improve the interface. Observations of user performance, for example, identified 36% of usability problems related to learning, 19% = screen design, 17% = terminology; 3% = system capabilities and 25% = other problems. Cross analysis of user semi-structured interviews pointed to the system’s ability to convey believable, visually appealing, realistic characters and classrooms. More contextual cues, multiple challenging behaviors featured at the same time, and changes to the visual appearance of the classroom would enhance realism. Revisions made to enhance the usability and instructional design elements of IVT-T are discussed. In addition, implications for teacher educators and researchers involved in the development of instructional technologies are summarized along with the potential value of including simulation in teacher training for behavior management.

[1]  Monique W. M. Jaspers,et al.  A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence , 2009, Int. J. Medical Informatics.

[2]  D. Kolb,et al.  Experiential Learning Theory: Previous Research and New Directions , 2000 .

[3]  Tim McLaughlin,et al.  A framework for evidence based visual style development for serious games , 2010, FDG.

[4]  D. Kaufman,et al.  Design Evaluation of a Simulation for Teacher Education , 2015 .

[5]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[6]  J. D. Fletcher.  Development of Professional Expertise: The Value of Expertise and Expert Performance: A Review of Evidence from the Military , 2009 .

[7]  Monica W. Tracey,et al.  The Instructional Design Knowledge Base: Theory, Research, and Practice , 2010 .

[8]  Mark A. Smylie,et al.  Expanding the Role of School Psychologists to Support Early Career Teachers: A Mixed-Method Study , 2016 .

[9]  Thomas A. Brush,et al.  Integrating technology into K-12 teaching and learning: current knowledge gaps and recommendations for future research , 2007 .

[10]  Luisa M. Regueras,et al.  Integration of an intelligent tutoring system in a course of computer network design , 2016, Educational Technology Research and Development.

[11]  H. Rex Hartson,et al.  Developing user interfaces: ensuring usability through product & process , 1993 .

[12]  Kraig Finstad.  The system usability scale and non-native English speakers , 2006 .

[13]  Robert A. Virzi,et al.  Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough? , 1992 .

[14]  Sue Gregory,et al.  VirtualPREX: Open and Distance Learning for pre-service teachers , 2011 .

[15]  Eleanor McLellan,et al.  Beyond the Qualitative Interview: Data Preparation and Transcription , 2003 .

[16]  Min Kyu Kim,et al.  Teacher professional development through digital content evaluation , 2017, Educational Technology Research and Development.

[17]  Chris Dede,et al.  Immersive Interfaces for Engagement and Learning , 2009, Science.

[18]  Elisa S. Shernoff,et al.  Redesigning community mental health services for urban children: Supporting schooling to promote mental health. , 2015, Journal of consulting and clinical psychology.

[19]  James Minogue,et al.  Safe science classrooms: Teacher training through serious educational games , 2014, Inf. Sci..

[20]  Monica W. Tracey,et al.  Instructional designers as reflective practitioners: developing professional identity through reflection , 2014 .

[21]  Stephen M. Alessi,et al.  Multimedia for Learning: Methods and Development , 2000 .

[22]  Doug A. Bowman,et al.  A Survey of Usability Evaluation in Virtual Environments: Classification and Comparison of Methods , 2002, Presence: Teleoperators & Virtual Environments.

[23]  A. Huberman,et al.  Qualitative Data Analysis: A Methods Sourcebook , 1994 .

[24]  Kara Dawson,et al.  A framework for aligning needs, abilities and affordances to inform design and practice of educational technologies , 2017, Br. J. Educ. Technol..

[25]  Eeva Nygren,et al.  Science Spots AR: a platform for science learning games with augmented reality , 2016, Educational Technology Research and Development.

[26]  Johnny Saldaña,et al.  The Coding Manual for Qualitative Researchers , 2009 .

[27]  Elisa S. Shernoff,et al.  Coaching Early Career Teachers in Urban Elementary Schools: A Mixed-Method Study , 2015 .

[28]  J. Smyth,et al.  Designing Serious Game Interventions for Individuals with Autism , 2014, Journal of Autism and Developmental Disorders.

[29]  Sebastiano Bagnara,et al.  Multimedia systems in distance education: effects of usability on learning , 1999, Interact. Comput..

[30]  K. Bierman,et al.  The impact of classroom aggression on the development of aggressive behavior problems in children , 2006, Development and Psychopathology.

[31]  Nik Thompson,et al.  Genetics with Jean: the design, development and evaluation of an affective tutoring system , 2017 .

[32]  John Doucette,et al.  Adopting electronic medical records in primary care: Lessons learned from health information systems implementation experience in seven countries , 2009, Int. J. Medical Informatics.

[33]  Lisa Abrams,et al.  Potential of one-to-one technologies in the classroom: teachers and students weigh in , 2017 .

[34]  Jennifer L. Frank,et al.  Using Virtual Role-Play to Enhance Teacher Candidates' Skills in Responding to Bullying. , 2017 .

[35]  Elisa S. Shernoff,et al.  A Qualitative Study of the Sources and Impact of Stress Among Urban Teachers , 2011 .

[36]  Linda S. Lotto.  Qualitative Data Analysis: A Sourcebook of New Methods , 1986 .

[37]  Charles E. Hughes,et al.  Virtual Classrooms: STAR Simulator Building Virtual Environments for Teacher Training in Effective Classroom Management. , 2007 .

[38]  D. Brunsma.  Uniforms in Public Schools: A Decade of Research and Debate. , 2005 .

[39]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[40]  Richard M. Ingersoll,et al.  The Impact of Induction and Mentoring Programs for Beginning Teachers , 2011 .

[41]  J. Edward Swan,et al.  Usability Engineering for Augmented Reality: Employing User-Based Studies to Inform Design , 2008, IEEE Transactions on Visualization and Computer Graphics.

[42]  S. Loeb,et al.  Teacher Churning , 2017 .

[43]  Chad Trevitte,et al.  The Rice University Press Initiative: An Interview with Charles Henry. , 2007 .

[44]  B. Lignugaris/Kraft,et al.  Meaningful Practice , 2017 .

[45]  Cassandra M. Guarino,et al.  Teacher Recruitment and Retention: A Review of the Recent Empirical Literature , 2006 .

[46]  Elisa S. Shernoff,et al.  Teacher Job Stress and Satisfaction in Urban Schools: Disentangling Individual-, Classroom-, and Organizational-Level Influences. , 2017, Behavior therapy.

[47]  Marc Erich Latoschik,et al.  Breaking Bad Behaviors: A New Tool for Learning Classroom Management Using Virtual Reality , 2016, Front. ICT.

[48]  Joseph L. Gabbard,et al.  Early Career Teacher Professional Development:Bridging Simulation Technology with Evidence-Based Behavior Management , 2018 .

[49]  Monika Akbar,et al.  Digital Technology Shaping Teaching Practices in Higher Education , 2016, Front. ICT.

[50]  Rex Hartson,et al.  The UX book, process and guidelines for ensuring a quality user experience by Rex Hartson and Pardha S. Pyla , 2012, SOEN.

[51]  Matthew B. Miles,et al.  Qualitative Data Analysis: An Expanded Sourcebook , 1994 .

[52]  Abdelkamel Tari,et al.  Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis , 2016, Comput. Hum. Behav..

[53]  M. Boermeester,et al.  Training situational awareness to reduce surgical errors in the operating room , 2015, The British journal of surgery.

[54]  Laura M. Desimone,et al.  Effects of Professional Development on Teachers’ Instruction: Results from a Three-year Longitudinal Study , 2002 .

[55]  Eli Lieber,et al.  Mixing Qualitative and Quantitative Methods: Insights into Design and Analysis Issues. , 2009 .

[56]  Carolyn M. Evertson,et al.  Handbook of Classroom Management: Research, Practice, and Contemporary Issues , 2012 .

[57]  Charles M. Reigeluth,et al.  Investigating “The Coolest School in America”: how technology is used in a learner-centered school , 2016 .

[58]  Melissa Gresalfi,et al.  Designing feedback in an immersive videogame: supporting student mathematical engagement , 2016 .

[59]  Traci Sitzmann.  A META-ANALYTIC EXAMINATION OF THE INSTRUCTIONAL EFFECTIVENESS OF COMPUTER-BASED SIMULATION GAMES , 2011 .

[60]  Lynne Cooke,et al.  Assessing Concurrent Think-Aloud Protocol as a Usability Test Method: A Technical Communication Approach , 2010, IEEE Transactions on Professional Communication.

[61]  Deborah Hix,et al.  User-Centered Design and Evaluation of Virtual Environments , 1999, IEEE Computer Graphics and Applications.

[62]  Christine L. Lisetti,et al.  On the Simulation of Human Frailty , 2011, BICA.

[63]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[64]  A. Fluck,et al.  Pre-service teachers’ perceptions of simSchool as preparation for inclusive education: a pilot study , 2014 .

[65]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[66]  David Gibson,et al.  simSchool: The game of teaching , 2005 .

[67]  L. Beutler,et al.  Virtual reality in psychotherapy training. , 2004, Journal of clinical psychology.

[68]  David Henry,et al.  Normative Influences on Aggression in Urban Elementary School Classrooms , 2000, American journal of community psychology.

[69]  Deborah J. Mayhew,et al.  The usability engineering lifecycle , 1999, CHI Extended Abstracts.

[70]  Julie Sarno Owens,et al.  Examining correlates of teacher receptivity to social influence strategies within a school consultation relationship , 2018, Psychology in the Schools.

[71]  George Sugai,et al.  Evidence-based Practices in Classroom Management: Considerations for Research to Practice , 2008 .

[72]  Marsha E Fonteyn,et al.  Developing a codebook to guide content analysis of expressive writing transcripts. , 2008, Applied nursing research : ANR.

[73]  LeeAnn Lindsey,et al.  Experiential Approach to Instruction , 2009 .

[74]  Catherine P. Bradshaw,et al.  Reducing Behavior Problems Among Students with Autism Spectrum Disorder: Coaching Teachers in a Mixed-Reality Setting , 2016, Journal of autism and developmental disorders.

[75]  J. Schepers,et al.  Stability of and Associations Between Social-Cognitive Determinants Over Time , 2015 .

[76]  Thomas R. Kratochwill,et al.  Transporting an evidence-based classroom management program for preschoolers with disruptive behavior problems to a school: An analysis of implementation, outcomes, and contextual variables. , 2007 .

[77]  Alessandro De Gloria,et al.  Designing Effective Serious Games: Opportunities and Challenges for Research , 2010, Int. J. Emerg. Technol. Learn..