Predicting expert-novice performance as serious games analytics with objective-oriented and navigational action sequences

New metrics are needed to produce insights in serious games analytics.Action sequences can be coded using task- or tile-based approach.Size of map grids can affects the accuracy in performance prediction.Grid size affects tile-based coding of action sequences and performance prediction.The optimal map grid size must be determined independently for each game. Previous research differentiated expert vs. novice performances based on how (dis)similar novices' action sequences were from that of the expert's by way of similarity measures. Action sequences were coded using an 'objective-oriented' (or task-based) approach based on the sequence of objectives/tasks completed in-game. Findings from these studies suggest that the task-based similarity measures is a better predictor than (a) distance traversed, and (b) time (of completion).In this study, we suggested an alternative method to code action sequences of experts and novices by way of a 'navigational' (or tile-based) approach. We divided a game-map into grids/tiles of different sizes to facilitate tracing of the path traversed by players in game and proceeded to test the effect of grid sizes on differentiating between experts and novices. We further compared the two different action sequence coding approaches and their abilities to measure players' competency improvement in serious games. The results of the study showed that the size of game grids does matter, and that both task-based and tile-based action sequence coding approaches are useful for serious games analytics.

[1]  Christopher D. Wickens,et al.  The Effects of Stress on Pilot Judgment in a MIDIS Simulator , 1993 .

[2]  Ruck Thawonmas,et al.  Analysis of revisitations in online games , 2011, Entertain. Comput..

[3]  Jérôme Dupire,et al.  Entertainment Computing – ICEC 2009 , 2009, Lecture Notes in Computer Science.

[4]  David L. Sheinberg,et al.  The Training and Transfer of Real-World Perceptual Expertise , 2005, Psychological science.

[5]  Ruck Thawonmas,et al.  Analysis of Area Revisitation Patterns in World of Warcarft , 2009, ICEC.

[6]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[7]  Monica W. Tracey,et al.  Does media affect learning: where are we now? , 2005 .

[8]  Douglas Landsittel,et al.  Can video games be used to predict or improve laparoscopic skills? , 2005, Journal of endourology.

[9]  Richard E. Clark,et al.  Confounding in Educational Computing Research , 1985 .

[10]  M. D. Dunnette.  Handbook of Industrial and Organizational Psychology , 2005 .

[11]  R. Clark.  Media will never influence learning , 1994 .

[12]  Byron Ellis.  Real-Time Analytics: Techniques to Analyze and Visualize Streaming Data , 2014 .

[13]  Bill Kapralos,et al.  Serious games for knee replacement surgery procedure education and training , 2010 .

[14]  Christian Sebastian Loh,et al.  Maximum Similarity Index (MSI): A metric to differentiate the performance of novices vs. multiple-experts in serious games , 2014, Comput. Hum. Behav..

[15]  Christoph Hölscher,et al.  Web search behavior of Internet experts and newbies , 2000, Comput. Networks.

[16]  R. Pieters,et al.  Visual attention during brand choice : The impact of time pressure and task motivation , 1999 .

[17]  M. Frese,et al.  Action as the core of work psychology: A German approach. , 1994 .

[18]  Dirk Ifenthaler,et al.  Assessment in Game-Based Learning: Foundations, Innovations, and Perspectives , 2012 .

[19]  Steven A. Hillyard,et al.  Neural Basis of Superior Performance of Action Videogame Players in an Attention-Demanding Task , 2011, The Journal of Neuroscience.

[20]  Karen Blackmore,et al.  A meta-analysis of data collection in serious games research , 2015 .

[21]  Christian Sebastian Loh,et al.  Performance metrics for serious games: Will the (real) expert please step forward? , 2013, Proceedings of CGAMES'2013 USA.

[22]  Georg Zoeller.  Game Development Telemetry in Production , 2013, Game Analytics, Maximizing the Value of Player Data.

[23]  Lowe Bryan William,et al.  Studies on the telegraphic language: The acquisition of a hierarchy of habits. , 1899 .

[24]  Jean Scholtz,et al.  Characteristics of the Mental Representations of Novice and Expert Programmers: An Empirical Study , 1993, Int. J. Man Mach. Stud..

[25]  Dirk Ifenthaler,et al.  Bridging the Gap between Expert-Novice Differences , 2010 .

[26]  D. Crookall.  Serious Games, Debriefing, and Simulation/Gaming as a Discipline , 2010 .

[27]  Dirk Ifenthaler,et al.  Serious games analytics : methodologies for performance measurement, assessment, and improvement , 2015 .

[28]  A Mark Williams,et al.  Perceptual-cognitive expertise in sport: some considerations when applying the expert performance approach. , 2005, Human movement science.

[29]  Christian Sebastian Loh,et al.  Information Trails: In-Process Assessment of Game-Based Learning , 2012 .

[30]  R. Clark,et al.  Dangers in the evaluation of instructional media , 1992, Academic medicine : journal of the Association of American Medical Colleges.

[31]  D. Ballabio,et al.  Classification tools in chemistry. Part 1: linear models. PLS-DA , 2013 .

[32]  A. Williams,et al.  Perceptual skill in soccer: Implications for talent identification and development , 2000, Journal of sports sciences.

[33]  Christian Sebastian Loh.  Designing Online Games Assessment as : Information Trails , 2007 .

[34]  S. Dreyfus,et al.  A Five-Stage Model of the Mental Activities Involved in Directed Skill Acquisition , 1980 .

[35]  Alessandro Canossa,et al.  Evaluating motion: spatial user behaviour in virtual environments , 2011, Int. J. Arts Technol..

[36]  M. Tarr,et al.  Training ‘greeble’ experts: a framework for studying expert object recognition processes , 1998, Vision Research.

[37]  Chiara Francalanci,et al.  DashMash: A Mashup Environment for End User Development , 2011, ICWE.

[38]  Christian Sebastian Loh,et al.  Measuring the (dis-)similarity between expert and novice behaviors as serious games analytics , 2013, Education and Information Technologies.

[39]  S. Dreyfus.  The Five-Stage Model of Adult Skill Acquisition , 2004 .

[40]  Julian Togelius,et al.  Game Data Mining , 2013, Game Analytics, Maximizing the Value of Player Data.

[41]  Susan T. Dumais,et al.  Large scale analysis of web revisitation patterns , 2008, CHI.

[42]  M. Stella Atkins,et al.  Eye gaze patterns differentiate novice and experts in a virtual laparoscopic surgery training environment , 2004, ETRA.

[43]  Cleotilde Gonzalez,et al.  Impact of numerical and graphical formats on dynamic decision making performance: an eye-tracking study , 2003, CLIHC '03.

[44]  Cleotilde González.  Decision Support for Real-Time, Dynamic Decision-Making Tasks , 2005 .

[45]  Jdm Underwood,et al.  Novice and expert performance with a dynamic control task: scanpaths during a computer game , 2005 .

[46]  Thomas H. Carr,et al.  From novice to expert performance : Memory, attention and the control of complex sensori-motor skills , 2004 .

[47]  Christian Sebastian Loh.  Improving the Impact and Return of Investment of Game-Based Learning , 2013, Int. J. Virtual Pers. Learn. Environ..

[48]  Ed H. Chi,et al.  Using information scent to model user information needs and actions and the Web , 2001, CHI.

[49]  Arnond Anantachai,et al.  ASSESSING WHAT PLAYERS LEARNED IN SERIOUS GAMES: IN SITU DATA COLLECTION, INFORMATION TRAILS, AND QUANTITATIVE ANALYSIS , 2007 .

[50]  Issa M. Khalil,et al.  Prediction of User's Web-Browsing Behavior: Application of Markov Model , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[51]  Michael Mateas,et al.  A data mining approach to strategy prediction , 2009, 2009 IEEE Symposium on Computational Intelligence and Games.

[52]  Matthias Rauterberg,et al.  From Novice to Expert Decision Behaviour: a Qualitative Modelling Approach with Petri Nets. , 1995 .

[53]  Steven C. Sutherland,et al.  Individual Differences in Causal Judgment under Time Pressure: Sex and Prior Video Game Experience as Predictors , 2011, International Journal of Comparative Psychology.

[54]  K. A. Ericsson,et al.  The making of an expert. , 2007, Harvard business review.

[55]  A. D. D. Groot.  Thought and Choice in Chess , 1978 .

[56]  Riccardo Berta,et al.  Assessment in and of Serious Games: An Overview , 2013, Adv. Hum. Comput. Interact..

[57]  S. Dreyfus,et al.  Peripheral Vision , 2005 .

[58]  Kyung-Eun Park,et al.  An Exploratory Study about Inaccuracy and Invalidity in Adolescent Self-Report Surveys , 2006 .

[59]  H. Zur,et al.  The effect of time pressure on risky choice behavior , 1981 .

[60]  James W. Tanaka,et al.  Learning to Become an Expert: Reinforcement Learning and the Acquisition of Perceptual Expertise , 2009, Journal of Cognitive Neuroscience.

[61]  George E. P. Box,et al.  Empirical Model‐Building and Response Surfaces , 1988 .

[62]  Eelco Herder.  Revisitation Patterns and Disorientation , 2003 .

[63]  D. Simons,et al.  The effects of video game playing on attention, memory, and executive control. , 2008, Acta psychologica.

[64]  R. Kozma,et al.  Multimedia and understanding: Expert and novice responses to different representations of chemical phenomena , 1997 .

[65]  Magy Seif El-Nasr,et al.  Game Analytics , 2013, Springer London.

[66]  Jon-Chao Hong,et al.  A study on thinking strategy between experts and novices of computer games , 2003, Comput. Hum. Behav..