Action-Based Learning Assessment in Virtual Training Environments

Action-based learning plays an important role in today’s education, with assessment being an inevitable aspect that presently requires extensive financial and human expert resources. A major research area is dedicated to the development and implementation of systems to perform an automated or at least support formative assessment of human actions in action-based learning. That is, recognising and comparing the performed actions of the learner with the previously recorded actions undertaken by experts. Human actions are classified according to the developed taxonomy of human actions to facilitate a formative feedback for the learner. The action-based learning assessment method analyses the action sequences of the learners according to reference solutions (generated by experts) and automatically provides formative feedback to improve learner performance.

[1]  V. Shute.  Focus on Formative Feedback , 2008 .

[2]  Gwo-Jen Hwang,et al.  A formative assessment-based mobile learning approach to improving the learning attitudes and achievements of students , 2011, Comput. Educ..

[3]  E. A. Fleishman,et al.  Toward a Taxonomy of Human Performance. , 1975 .

[4]  Harry Torrance,et al.  Evaluating authentic assessment : problems and possibilities in new approaches to assessment , 1995 .

[5]  Amr Ahmed,et al.  Affordable interactive virtual reality system for the Dynamic Hip Screw surgery training in vitro , 2012 .

[6]  Xudong Wang,et al.  A Virtual Training System Using a Force Feedback Haptic Device for Oral Implantology , 2012, Trans. Edutainment.

[7]  Torsten Reiners,et al.  The Taxonomy of Goal-oriented Actions in Virtual Training Environments , 2014 .

[8]  J. Grimshaw,et al.  What is an adequate sample size? Operationalising data saturation for theory-based interview studies , 2010, Psychology & health.

[9]  Kenneth D. Bailey,et al.  Typologies And Taxonomies , 1994 .

[10]  Thomas C. Reeves,et al.  A Guide to Authentic e-Learning , 2009 .

[11]  Björn Niehaves,et al.  Evaluation of Conceptual Models - A Structuralist Approach , 2005, ECIS.

[12]  Mustafa Alshawi,et al.  Construction industry offsite production: A virtual reality interactive training environment prototype , 2012, Adv. Eng. Informatics.

[13]  Angélica de Antonio Jiménez,et al.  Steve Meets Jack: The Integration of an Intelligent Tutor and a Virtual Environment with Planning Capabilities , 2003, IVA.

[14]  D. Boud.  Assessment and learning: contradictory or complementary? , 1995 .

[15]  Guy H. Walker,et al.  Human factors design methods review , 2003 .

[16]  S J Grobe,et al.  The conceptual validity of a taxonomy of nursing interventions. , 1993, Journal of advanced nursing.

[17]  Jorma Tarhio,et al.  ACE: Automated Compiler Exercises , 2004 .

[18]  Juhani Iivari,et al.  The IS Core - VII: Towards Information Systems as a Science of Meta-Artifacts , 2003, Commun. Assoc. Inf. Syst..

[19]  Peter E. Smith.  Simulating the Classroom with Media and Computers , 1987 .

[20]  Sasha A. Barab,et al.  A co-evolutionary model for supporting the emergence of authenticity , 2000 .

[21]  Vicki L. Schmitt,et al.  Coming to Terms With Classroom Assessment , 2007 .

[22]  Joseph J. Schwab.  Inquiry, the Science Teacher, and the Educator , 1960, The School Review.

[23]  D. Bell,et al.  The Sage Encyclopedia of Qualitative Research Methods , 2013 .

[24]  M. Taras,et al.  ASSESSMENT – SUMMATIVE AND FORMATIVE – SOME THEORETICAL REFLECTIONS , 2005 .

[25]  Jeffrey D. Wilhelm,et al.  Inquiring Minds Learn to Read, Write, and Think: Reaching All Learners through Inquiry , 2010 .

[26]  Robert Williams,et al.  Automatically grading essays with markit , 2004 .

[27]  Jan Pries-Heje,et al.  The Design Theory Nexus , 2008, MIS Q..

[28]  Thomas C. Reeves,et al.  Authentic activities and online learning , 2002 .

[29]  Ahmad Alturki,et al.  A Design Science Research Roadmap , 2011, PACIS.

[30]  D. Allen,et al.  What is Problem-Based Learning? , 2006 .

[31]  C. Mills.  Situated Actions and Vocabularies of Motive , 1940 .

[32]  Edward Errington,et al.  Developing Scenario-based Learning: practical insights for tertiary educators , 2003 .

[33]  D. Sadler.  Formative assessment and the design of instructional systems , 1989 .

[34]  J. W. Gikandi,et al.  Online formative assessment in higher education: A review of the literature , 2011, Comput. Educ..

[35]  P. Jaccard.  THE DISTRIBUTION OF THE FLORA IN THE ALPINE ZONE.1 , 1912 .

[36]  Eileen Trotter.  Student perceptions of continuous summative assessment , 2006 .

[37]  N. Nagelkerke,et al.  A note on a general definition of the coefficient of determination , 1991 .

[38]  Larry Ambrose,et al.  The power of feedback. , 2002, Healthcare executive.

[39]  Toni Robertson,et al.  Building bridges: negotiating the gap between work practice and technology design , 2000, Int. J. Hum. Comput. Stud..

[40]  Richard W. Obermayer,et al.  Human performance prediction in man-machine systems. Volume 1 - A technical review , 1970 .

[41]  Mehmet S. Kizil,et al.  Virtual reality applications in Australian minerals industry , 2003 .

[42]  Norman I. Badler,et al.  A taxonomy and comparison of haptic actions for disassembly tasks , 2003, IEEE Virtual Reality, 2003. Proceedings..

[43]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[44]  Valerie J. Shute,et al.  A Large-Scale Evaluation of an Intelligent Discovery World: Smithtown , 1990, Interact. Learn. Environ..

[45]  W. Lewis Johnson,et al.  STEVE (video session): a pedagogical agent for virtual reality , 1998, AGENTS '98.

[46]  B. Wilson.  Constructivist Learning Environments: Case Studies in Instructional Design , 1995 .

[47]  Peter Loos,et al.  Multiperspective Evaluation of Reference Models - Towards a Framework , 2003, ER.

[48]  Darren George,et al.  IBM SPSS Statistics 21 Step by Step: A Simple Guide and Reference , 2013 .

[49]  K. Kuutti.  Activity theory as a potential framework for human-computer interaction research , 1995 .

[50]  U. Cress,et al.  Training in virtual environments: putting theory into practice , 2013, Ergonomics.

[51]  Simeon J. Simoff,et al.  Virtual Agents and 3D Virtual Worlds for Preserving and Simulating Cultures , 2009, IVA.

[52]  D. Shwayder.  The Stratification of behaviour , 1965 .

[53]  Thomas C. Reeves,et al.  Patterns of engagement in authentic online learning environments , 2003, ASCILITE.

[54]  Vicki L. Schmitt,et al.  Defining Authentic Classroom Assessment. , 2012 .

[55]  Michael D. Mumford,et al.  EVALUATING CLASSIFICATIONS OF JOB BEHAVIOR: A CONSTRUCT VALIDATION OF THE ABILITY REQUIREMENT SCALES , 2006 .

[56]  Danielle Mirliss,et al.  Incident and Disaster Management Training: Collaborative Learning Opportunities Using Virtual World Scenarios , 2010 .

[57]  Brian C. Nelson,et al.  Visual signaling in virtual world-based assessments: The SAVE Science project , 2014, Inf. Sci..

[58]  Leslie Jarmon,et al.  Virtual world teaching, experiential learning, and assessment: An interdisciplinary communication course in Second Life , 2009, Comput. Educ..

[59]  Sue Gregory,et al.  Real thinking with virtual hats: A role-playing activity for pre-service teachers in Second Life , 2012 .

[60]  E. Guba,et al.  Paradigmatic Controversies, Contradictions, and Emerging Confluences. , 2005 .

[61]  James F. Allen.  Towards a General Theory of Action and Time , 1984, Artif. Intell..

[62]  G. Veletsianos,et al.  A review of adventure learning , 2009 .

[63]  mc schraefel,et al.  A Taxonomy of Gestures in Human Computer Interactions , 2005 .

[64]  Jay McTighe,et al.  Seven Practices for Effective Learning , 2005 .

[65]  Stanley Lemeshow,et al.  Sampling of Populations: Methods and Applications , 1991 .

[66]  P. Ekman,et al.  The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding , 1969 .

[67]  A. Doering.  Adventure Learning: Transformative hybrid online education , 2006 .

[68]  L. Earl,et al.  Assessment as Learning: Using Classroom Assessment to Maximize Student Learning , 2003 .

[69]  V. Shute,et al.  Stealth Assessment: Measuring and Supporting Learning in Video Games , 2013 .

[70]  E Salas,et al.  Task and training requirements analysis methodology (TTRAM): an analytic methodology for identifying potential training uses of simulator networks in teamwork-intensive task environments. , 1998, Ergonomics.

[71]  K. M. Breakey,et al.  The Use of Scenario-Based-Learning Interactive Software to Create Custom Virtual Laboratory Scenarios for Teaching Genetics , 2008, Genetics.

[72]  A. Herrington,et al.  What is an authentic learning environment , 2006 .

[73]  Anton J. Kuzel,et al.  Sampling in qualitative inquiry. , 1992 .

[74]  Christian Gütl,et al.  Modular Assessment System for Modern Learning Settings: MASS , 2010 .

[75]  Eleni Stroulia,et al.  A framework for monitoring instructional environments in a virtual world , 2014, Br. J. Educ. Technol..

[76]  Deva Ramanan,et al.  Detecting activities of daily living in first-person camera views , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[77]  Bruno Arnaldi,et al.  Scenario sharing in a collaborative virtual environment for training , 2008, VRST '08.

[78]  Salvatore Valenti,et al.  An Overview of Current Research on Automated Essay Grading , 2003, J. Inf. Technol. Educ..

[79]  Rama Chellappa,et al.  Machine Recognition of Human Activities: A Survey , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[80]  W. Neuman,et al.  Social Research Methods: Qualitative and Quantitative Approaches , 2002 .

[81]  C. Rogers.  Client-Centered Therapy: Its Current Practice, Implications and Theory , 1951 .

[82]  A. M. Novikov,et al.  Research Methodology: From Philosophy of Science to Research Design , 2013 .

[83]  Meredith Ringel Morris,et al.  User-defined gestures for surface computing , 2009, CHI.

[84]  Wondea Jung,et al.  A taxonomy of performance influencing factors for human reliability analysis of emergency tasks , 2003 .

[85]  H. Dreher,et al.  Automated essay grading systems applied to a first year university subject: how can we do it better? , 2002 .

[86]  John R. Kender,et al.  Annotation and taxonomy of gestures in lecture videos , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[87]  C. Abraham,et al.  The Behavior Change Technique Taxonomy (v1) of 93 Hierarchically Clustered Techniques: Building an International Consensus for the Reporting of Behavior Change Interventions , 2013, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[88]  Tzu-Chien Liu,et al.  Supporting Teachers in Identifying Students' Learning Styles in Learning Management Systems: An Automatic Student Modelling Approach , 2009, J. Educ. Technol. Soc..

[89]  Christian Sebastian Loh,et al.  Maximum Similarity Index (MSI): A metric to differentiate the performance of novices vs. multiple-experts in serious games , 2014, Comput. Hum. Behav..

[90]  Andy Cockburn,et al.  User-defined gestures for augmented reality , 2013, INTERACT.

[91]  Catherine Pelachaud,et al.  Rules for Responsive Robots: Using Human Interactions to Build Virtual Interactions , 2002 .

[92]  Klaus Zechner,et al.  Automated Essay Scoring: Writing Assessment and Instruction , 2010 .

[93]  William R. Cockayne.  Two-hand, whole-hand interaction , 1998 .

[94]  Torsten Reiners,et al.  Investigating Factors Affecting the Uptake of Automated Assessment Technology , 2011, J. Inf. Technol. Educ. Res..

[95]  J. Cacioppo,et al.  Inferring psychological significance from physiological signals. , 1990, The American psychologist.

[96]  M. Segers,et al.  Effects of problem-based learning: a meta- analysis , 2003 .

[97]  Mark Tucker,et al.  Designing and Implementing an Assessment Plan for a Virtual Engineering Lab , 2010 .

[98]  D. Mertens.  Research and Evaluation in Education and Psychology: Integrating Diversity with Quantitative, Qualitative, and Mixed Methods , 1997 .

[99]  Laurie A. Broedling,et al.  Taxonomies of Human Performance: The Description of Human Tasks , 1984 .

[100]  Colin Beard,et al.  Experiential Learning: A Handbook for Education, Training and Coaching , 2013 .

[101]  A. Dores,et al.  Computer-Assisted Rehabilitation Program – Virtual Reality (CARP-VR): A Program for Cognitive Rehabilitation of Executive Dysfunction , 2011 .

[102]  V. Shute,et al.  Melding the Power of Serious Games and Embedded Assessment to Monitor and Foster Learning: Flow and Grow , 2009 .

[103]  Jacqueline Urakami,et al.  Developing and Testing a Human-Based Gesture Vocabulary for Tabletop Systems , 2012, Hum. Factors.

[104]  Michalis Vazirgiannis,et al.  Cluster validity methods: part I , 2002, SGMD.

[105]  David L. Davidson,et al.  The Logical Form of Action Sentences , 2001 .

[106]  Barney Dalgarno,et al.  Changing directions through VirtualPREX: engaging pre-service teachers in virtual professional experience , 2011 .

[107]  Benny P. L. Lo,et al.  Blurring the boundaries: scenario‐based simulation in a clinical setting , 2005, Medical education.

[108]  Heinz Dreher,et al.  Towards Developing Automated Assessment in Virtual Learning Environments , 2012 .

[109]  Sue Swaffield,et al.  Getting to the heart of authentic Assessment for Learning , 2011 .

[110]  Jennifer Tichon,et al.  Using Presence to Improve a Virtual Training Environment , 2007, Cyberpsychology Behav. Soc. Netw..

[111]  I. Clark.  Formative Assessment: Assessment Is for Self-regulated Learning , 2012, Educational Psychology Review.

[112]  M. Herron,et al.  The Nature of Scientific Enquiry , 1971, The School Review.

[113]  Paul Black,et al.  Assessment for Learning in the Classroom , 2004 .

[114]  Tim S. Roberts.  The use of multiple choice tests for formative and summative assessment , 2006 .

[115]  Autumn B. Hostetter,et al.  Visible embodiment: Gestures as simulated action , 2008, Psychonomic bulletin & review.

[116]  Alan R. Hevner,et al.  The Three Cycle View of Design Science , 2007, Scand. J. Inf. Syst..

[117]  Jacquelyn Ford Morie,et al.  Performing in (virtual) spaces: Embodiment and being in virtual environments , 2007 .

[118]  Stephen C. Bronack,et al.  Designing Virtual Worlds to Facilitate Meaningful Communication : Issues , Considerations , and Lessons Learned , 2008 .

[119]  Torsten Reiners,et al.  Immersive Virtual Environments to Facilitate Authentic Education in Logistics and Supply Chain Management , 2013 .

[120]  Christer Carlsson,et al.  DIVE - A platform for multi-user virtual environments , 1993, Comput. Graph..

[121]  Christian Gütl,et al.  NDive: The story of how logistics and supply chain management could be taught , 2013 .

[122]  Blake Hannaford,et al.  A Hybrid Discriminative/Generative Approach for Modeling Human Activities , 2005, IJCAI.

[123]  Juhani Iivari,et al.  A Paradigmatic Analysis of Information Systems As a Design Science , 2007, Scand. J. Inf. Syst..

[124]  Russell G. Almond,et al.  A cognitive task analysis with implications for designing simulation-based performance assessment☆ , 1999 .

[125]  B. Belland.  The Role of Construct Definition in the Creation of Formative Assessments in Game-Based Learning , 2012 .

[126]  K. Perreault,et al.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2011 .

[127]  Peter Twining,et al.  Exploring the educational potential of virtual worlds - Some reflections from the SPP , 2009, Br. J. Educ. Technol..

[128]  Mary Beth Rosson,et al.  Scenario-based design , 2002 .

[129]  Myron W. Krueger,et al.  Artificial reality II , 1991 .

[130]  Ross E. Willis,et al.  A multicenter, simulation-based skills training collaborative using shared GI mentor II systems: results from the Texas association of surgical skills laboratories (TASSL) flexible endoscopy curriculum , 2011, Surgical Endoscopy.

[131]  David M. Shannon,et al.  Using Electronic Surveys: Advice from Survey Professionals. , 2002 .

[132]  Mark W. Bell.  Toward a Definition of “Virtual Worlds” , 1970 .

[133]  Ronan Querrec,et al.  MASCARET: pedagogical multi-agents systems for virtual environment for training , 2003, Proceedings. 2003 International Conference on Cyberworlds.

[134]  Keng Siau,et al.  3-D Virtual Worlds in Education: Applications, Benefits, Issues, and Opportunities , 2008, J. Database Manag..

[135]  Sandeep Purao,et al.  Design Research in the Technology of Information Systems: Truth or Dare , 2002 .

[136]  Gonzalo Méndez,et al.  PRVIR : An Integration between an Intelligent Tutoring System and a Virtual Environment , 2001 .

[137]  Richard O. Peterson,et al.  DERIVING TRAINING DEVICE IMPLICATIONS FROM LEARNING THEORY PRINCIPLES. VOLUME II: METHODOLOGY , 1961 .

[138]  Roy Rada,et al.  Modelling Situated Actions in Collaborative Hypertext Databases. , 2006 .

[139]  Jan Pries-Heje,et al.  Strategies for Design Science Research Evaluation , 2008, ECIS.

[140]  Dietrich Albert,et al.  Applying a Web and Simulation-Based System for Adaptive Competence Assessment of Spinal Anaesthesia , 2009, ICWL.

[141]  Nicholas Dronen,et al.  Automated feedback in a large-scale implementation of a formative writing system: Implications for improving student , 2014 .

[142]  Laura Johnson,et al.  How Many Interviews Are Enough? , 2006 .

[143]  Jay F. Nunamaker,et al.  Systems Development in Information Systems Research , 1990, J. Manag. Inf. Syst..

[144]  Philippe Fuchs.  Introduction to virtual reality , 2011 .

[145]  Bruno Arnaldi,et al.  Virtual Environments for Training: From Individual Learning to Collaboration with Humanoids , 2007, Edutainment.

[146]  Mien Segers,et al.  New Assessment Forms in Problem-based Learning: The value-added of the students' perspective , 2001 .

[147]  Robert L. Winkler,et al.  Multiple Experts vs. Multiple Methods: Combining Correlation Assessments , 2004, Decis. Anal..

[148]  Alice Man Sze Lau,et al.  Assessment for learning in higher education , 2013 .

[149]  Jonathan Steuer,et al.  Defining virtual reality: dimensions determining telepresence , 1992 .

[150]  Jeff Rickel,et al.  Virtual Humans for Team Training in Virtual Reality , 1999 .

[151]  C. V. D. van der Vleuten,et al.  The mechanism of impact of summative assessment on medical students’ learning , 2010, Advances in health sciences education : theory and practice.

[152]  E. A. Attree,et al.  Training in virtual environments: transfer to real world tasks and equivalence to real task training , 2000, Ergonomics.

[153]  Roger C. Schank,et al.  Virtual Learning: A Revolutionary Approach to Building a Highly Skilled Workforce , 1997 .

[154]  John B. Kervin,et al.  Methods for Business Research , 1992 .

[155]  Mark A. McDaniel,et al.  Using quizzes to enhance summative-assessment performance in a web-based class: An experimental study , 2012 .

[156]  Lucy Suchman,et al.  Human-Machine Reconfigurations: Plans and Situated Actions , 2006 .

[157]  M. Scriven.  The methodology of evaluation , 1966 .

[158]  C. Mallows,et al.  A Method for Comparing Two Hierarchical Clusterings , 1983 .

[159]  P. Black,et al.  Assessment and Classroom Learning , 1998 .

[160]  Torsten Reiners,et al.  Game-Based Elements to Upgrade Bots to Non-Player Characters in Support of Educators , 2014 .

[161]  Martin Bichler,et al.  Design science in information systems research , 2006, Wirtschaftsinf..

[162]  M. Helfert,et al.  Evaluation Patterns for Design Science Research Artefacts , 2014 .

[163]  Theo J. Bastiaens,et al.  The surplus value of an authentic learning environment , 2005, Comput. Hum. Behav..

[164]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[165]  D. Gentner.  Structure‐Mapping: A Theoretical Framework for Analogy* , 1983 .

[166]  Marten Schönherr,et al.  Outline of a design science research process , 2009, DESRIST.

[167]  J. Aken.  Management Research Based on the Paradigm of the Design Sciences: The Quest for Field-Tested and Grounded Technological Rules , 2004 .

[168]  R. Clemen.  Combining forecasts: A review and annotated bibliography , 1989 .

[169]  Corwin A. Bennett,et al.  Toward Empirical, Practicable, Comprehensive Task Taxonomy , 1971 .

[170]  B. Bloom,et al.  Taxonomy of Educational Objectives. Handbook I: Cognitive Domain , 1966 .

[171]  Jan Herrington,et al.  Authentic Assessment and Multimedia: how university students respond to a model of authentic assessment , 1998 .

[172]  Toni Robertson,et al.  Cooperative Work and Lived Cognition: A Taxonomy of Embodied Actions , 1997, ECSCW.

[173]  N. Mackenzie,et al.  Research dilemmas: Paradigms, methods and methodology , 2006 .

[174]  John M. Carroll,et al.  Making Use: Scenario-Based Design of Human-Computer Interactions , 2000 .

[175]  Jacquelyn Ford Morie,et al.  Virtual Chironomia: Developing Standards for Non-verbal Communication in Virtual Worlds , 2009 .

[176]  Helen Farley.  Interoperability, learning designs and virtual worlds: issues and strategies , 2010 .

[177]  Vijay K. Vaishnavi,et al.  Theory Development in Design Science Research: Anatomy of a Research Project , 2008 .

[178]  N. Stanton,et al.  Rapid Assessment of Tasks & Context (RATaC): Methodological Development , 2009 .

[179]  Jan Herrington,et al.  Authentic Learning Environments In Higher Education , 2005 .

[180]  Torsten Reiners,et al.  Action-Based Learning Assessment Method (ALAM) in Virtual Training Environments , 2013 .

[181]  Craig W. Thompson,et al.  Modeling Healthcare Logistics in a Virtual World , 2008, IEEE Internet Computing.

[182]  Yves De Smet,et al.  Finding compromise solutions in project portfolio selection with multiple experts by inverse optimization , 2016, Comput. Oper. Res..

[183]  Samir Chatterjee,et al.  A Design Science Research Methodology for Information Systems Research , 2008 .

[184]  Birgit Kopainsky,et al.  Automated assessment of learners' understanding in complex dynamic systems , 2012 .

[185]  John R. Venable,et al.  The role of theory and theorising in Design Science research , 2006 .

[186]  T M Ruff,et al.  Virtual reality for mine safety training. , 2000, Applied occupational and environmental hygiene.

[187]  Lincoln C. Wood,et al.  Authentic, immersive and emotional experience in virtual learning environments: The fear of dying as an important learning experience in a simulation , 2014 .

[188]  Angélica de Antonio Jiménez,et al.  Intelligent Virtual Environments for Training in Nuclear Power Plants , 2004, ICEIS.

[189]  Torsten Reiners,et al.  How to Teach and Demonstrate Topics of Supply Chain Management in Virtual Worlds , 2008 .

[190]  Steven Hannah,et al.  Sorting Out Card Sorting: Comparing Methods for Information Architects, Usability Specialists, and Other Practitioners , 2008 .

[191]  Tuure Tuunanen,et al.  Design Science Research Evaluation , 2012, DESRIST.

[192]  Pierre Dillenbourg,et al.  Virtual Learning Environments , 2020, The SAGE Encyclopedia of Higher Education.

[193]  A. Duffy,et al.  Completion of a novel, virtual-reality-based, advanced laparoscopic curriculum improves advanced laparoscopic skills in senior residents. , 2011, Journal of surgical education.

[194]  James Ong.  Automated Performance Assessment and Feedback for Free-Play Simulation-Based Training. , 2007 .

[195]  Valerie J. Janesick.  Authentic Assessment Primer , 2006 .

[196]  George Gaylord Simpson,et al.  Principles of Animal Taxonomy , 1961 .

[197]  Vijay K. Vaishnavi,et al.  A Framework for Theory Development in Design Science Research: Multiple Perspectives , 2012, J. Assoc. Inf. Syst..

[198]  Mike Wu,et al.  A study of hand shape use in tabletop gesture interaction , 2006, CHI Extended Abstracts.

[199]  Kathleen Iverson,et al.  Scenario-based E-learning design , 2004 .

[200]  Brian Everitt,et al.  Detecting Clusters Graphically , 2011 .

[201]  Som Naidu.  Using scenario-based learning to promote situated learning and develop professional knowledge , 2010 .

[202]  John R. Savery,et al.  Overview of Problem-Based Learning: Definitions and Distinctions. , 2006 .

[203]  Chris Dede,et al.  Multisensory Immersion as a Modeling Environment for Learning Complex Scientific Concepts , 1999 .

[204]  Jacques Felblinger,et al.  The virtual reality simulator dV-Trainer® is a valid assessment tool for robotic surgical skills , 2012, Surgical Endoscopy.

[205]  Eduardo Salas,et al.  Development of a task information taxonomy for human performance systems. , 1990 .

[206]  A. Smith,et al.  Research Methodology: A Step-by-step Guide for Beginners , 2012 .

[207]  Chris Gunn,et al.  Collaborative virtual sculpting with haptic feedback , 2006, Virtual Reality.

[208]  P. Black,et al.  Developing the theory of formative assessment , 2009 .

[209]  Lisbeth Klastrup.  Towards a Poetics of Virtual Worlds: Multi-User Textuality and the Emergence of Story , 2003 .

[210]  D. Kolb.  Experiential Learning: Experience as the Source of Learning and Development , 1983 .

[211]  Roy D. Pea,et al.  Addressing the Challenges of Inquiry-Based Learning Through Technology and Curriculum Design , 1999 .

[212]  C. Chen,et al.  A virtual reality based fire training simulator with smoke hazard assessment capacity , 2014, Adv. Eng. Softw..

[213]  Meredith Ringel Morris,et al.  ShadowGuides: visualizations for in-situ learning of multi-touch and whole-hand gestures , 2009, ITS '09.

[214]  Rudolph P. Darken,et al.  The Application of Human Ability Requirements to Virtual Environment Interface Design and Evaluation , 2003 .

[215]  H. Barrows.  Problem‐based learning in medicine and beyond: A brief overview , 1996 .

[216]  Phillip M. Harter,et al.  Simulation for Team Training and Assessment: Case Studies of Online Training with Virtual Worlds , 2008, World Journal of Surgery.

[217]  Duncan David Nulty,et al.  The adequacy of response rates to online and paper surveys: what can be done? , 2008 .

[218]  J.K. Aggarwal,et al.  Human activity analysis , 2011, ACM Comput. Surv..

[219]  D. Kolb,et al.  Experiential Learning Theory , 2002 .

[220]  Shirley Gregor,et al.  The Anatomy of a Design Theory , 2007, J. Assoc. Inf. Syst..

[221]  Daniel C. Moos,et al.  Adventure Learning , 2011 .

[222]  David M. Shannon,et al.  A Comparison of Response Rate, Response Time, and Costs of Mail and Electronic Surveys , 2002 .

[223]  Som Naidu.  Instructional designs for distance education , 2007 .

[224]  L. Hunt.  Authentic Learning at Work , 2006 .

[225]  Jeff Craighead,et al.  Distributed, game-based, intelligent tutoring systems — The next step in computer based training? , 2008, 2008 International Symposium on Collaborative Technologies and Systems.

[226]  Ronei Marcos de Moraes,et al.  Skills Assessment of Users in Medical Training Based on Virtual Reality Using Bayesian Networks , 2012, CIARP.

[227]  R. Tamblyn,et al.  Problem-based learning : an approach to medical education , 1980 .

[228]  L. Roberts,et al.  Oral presentation as a form of summative assessment in a master’s level PGCE module: the student perspective , 2012 .

[229]  Hsu-Shih Shih,et al.  Recruitment and selection processes through an effective GDSS , 2005 .

[230]  G. Wiggins.  The Case for Authentic Assessment. , 1990 .

[231]  Ruimin Shen,et al.  The intelligent assessment system in Web-based distance learning education , 2001, 31st Annual Frontiers in Education Conference. Impact on Engineering and Science Education. Conference Proceedings (Cat. No.01CH37193).

[232]  Xinming Hu,et al.  Automated Assessment System for Subjective Questions Based on LSI , 2010, 2010 Third International Symposium on Intelligent Information Technology and Security Informatics.

[233]  Diane Hart.  Authentic Assessment: A Handbook for Educators. Assessment Bookshelf Series. , 1994 .

[234]  Torsten Reiners,et al.  New landscapes and new eyes: the role of virtual world design for supply chain education , 2014 .

[235]  N. Freedman,et al.  Kinetic Behavior in Altered Clinical States: Approach to Objective Analysis of Motor Behavior during Clinical Interviews , 1967, Perceptual and motor skills.

[236]  Barbara Lasky,et al.  Practising what we teach: vocational teachers learn to research through applying action learning techniques , 2004 .

[237]  Torsten Reiners,et al.  From Chaos Towards Sense: A Learner-Centric Narrative Virtual Learning Space , 2014 .

[238]  Dariush Nejad Ansari,et al.  Automated Versus Human Essay Scoring: A Comparative Study , 2012 .

[239]  Jonathan Evans,et al.  Rationality and reasoning , 1996 .

[240]  F. Dochy,et al.  Peer assessment in problem based learning , 2001 .

[241]  Keng Siau,et al.  Evaluation techniques for systems analysis and design modelling methods – a review and comparative analysis , 2011, Inf. Syst. J..

[242]  P. Kirschner,et al.  A five-dimensional framework for authentic assessment , 2004 .

[243]  Anne Cleven,et al.  Design alternatives for the evaluation of design science research artifacts , 2009, DESRIST.

[244]  Alan R. Hevner,et al.  Design Research in Information Systems: Theory and Practice , 2010 .

[245]  Erik Hollnagel,et al.  Failures without errors: quantification of context in HRA , 2004, Reliab. Eng. Syst. Saf..

[246]  A. Goldman.  Theory of Human Action , 1970 .

[247]  Adham Mackieh,et al.  Effects of performance shaping factors on human error , 1998 .

[248]  Sue Crane.  Integrated Science in a Restructured High School. , 1991 .

[249]  Mitchel Resnick,et al.  “Thick” authenticity: new media and authentic learning , 1999 .

[250]  Stefan Schutt,et al.  Virtual worlds in Australian and New Zealand higher education: remembering the past, understanding the present and imagining the future , 2013 .

[251]  John Annett.  Hierarchical Task Analysis (HTA) , 2004 .

[252]  Ulrich Frank,et al.  Evaluation of Reference Models , 2006 .

[253]  Jan Herrington,et al.  Establishing the critical elements that determine authentic assessment , 2014 .

[254]  Action-based learning , 2012 .

[255]  R. Stuart,et al.  Action Based “Learning”: Are Activity and Experience the Same? , 1987 .

[256]  António M. Lopes,et al.  A new type haptics-based virtual environment system for assembly training of complex products , 2011, The International Journal of Advanced Manufacturing Technology.

[257]  Alan Miller,et al.  A taxonomy of virtual worlds usage in education , 2012, Br. J. Educ. Technol..