Measuring learning in serious games: a case study with structural assessment

The effectiveness of serious games is often measured with verbal assessment. As an alternative we propose Pathfinder structural assessment (defined as measuring the learners’ knowledge organization and compare this with a referent structure) which comprises three steps: knowledge elicitation, knowledge representation and knowledge evaluation. We discuss practical and theoretical considerations for the use of structural assessment and showcase its application with the game Code Red: Triage. Results suggest that structural assessment measures an individual’s understanding of a domain at least differently from verbal assessment. While verbal assessment may provide a more nuanced picture regarding declarative and procedural knowledge, structural assessment may add an in-depth understanding of the concepts that are regarded important in a domain. In the Discussion we propose four guidelines to effectively use structural assessment in serious games: (1) Determine the appropriateness of the domain for structural assessment, (2) select an appropriate referent for the target group(s), (3) select the number of concepts needed for structural assessment, and (4) consider the analysis of the graphical knowledge representations to obtain in-depth information about the quality of the knowledge structures.

[1]  A. Biglan.  The characteristics of subject matter in different academic areas. , 1973 .

[2]  A. Biglan.  Relationships between subject matter characteristics and the structure and output of university departments. , 1973 .

[3]  B. Adelson.  Problem solving and the development of abstract categories in programming languages , 1981, Memory & cognition.

[4]  G. Diekhoff.  Testing Through Relationship Judgments. , 1983 .

[5]  Barbara S. Plake,et al.  Preface: Social and Technical Issues in Testing: Implications for Test Construction and Usage , 1984 .

[6]  M. Chi,et al.  The Nature of Expertise , 1988 .

[7]  David H. Jonassen,et al.  Designing structured hypertext and structuring access to hypertext , 1988 .

[8]  Robert J. Sternberg,et al.  Domain-Generality versus Domain-Specificity: The Life and Impending Death of a False Dichotomy. , 1989 .

[9]  Francis T. Durso,et al.  A measure of the knowledge reorganization underlying insight , 1990 .

[10]  Roger W. Schvaneveldt,et al.  Pathfinder associative networks: studies in knowledge organization , 1990 .

[11]  James E. McDonald,et al.  Hypertext perspectives: using pathfinder to build hypertext systems , 1990 .

[12]  Timothy E. Goldsmith,et al.  A structural assessment of classroom learning , 1990 .

[13]  Thomas J. Shuell,et al.  Phases of Meaningful Learning , 1990 .

[14]  Peder J. Johnson,et al.  Assessing Structural Knowledge. , 1991 .

[15]  Henk G. Schmidt,et al.  On the Role of Biomedical Knowledge in Clinical Reasoning by Experts, Intermediates and Novices , 1992, Cogn. Sci..

[16]  David H. Jonassen,et al.  Structural Knowledge: Techniques for Representing, Conveying, and Acquiring Structural Knowledge , 1993 .

[17]  E. Salas,et al.  Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. , 1993 .

[18]  José J. Cañas,et al.  Structural Representations in Knowledge Acquisition. , 1994 .

[19]  Timothy E. Goldsmith,et al.  Structural knowledge assessment: comparison of referent structures , 1994 .

[20]  Eduardo Salas,et al.  Measuring Knowledge Organization as a Method for Assessing Learning during Training , 1995, Hum. Factors.

[21]  Gwendolyn E. Campbell,et al.  Assessing Knowledge Structures: Relations With Experience and Posttraining Performance , 1999 .

[22]  W C McGaghie,et al.  Quantitative concept mapping in pulmonary physiology: comparison of student and faculty knowledge structures. , 2000, Advances in physiology education.

[23]  E. Day,et al.  Knowledge structures and the acquisition of a complex skill. , 2001, The Journal of applied psychology.

[24]  Shai Simonson,et al.  Assessing knowledge change in computer science , 2006, Comput. Sci. Educ..

[25]  J. Michael Spector,et al.  Highly integrated model assessment technology and tools , 2010, CELDA 2008.

[26]  Jeroen Keppens,et al.  Concept map assessment for teaching computer programming , 2008, Comput. Sci. Educ..

[27]  Thomas Connolly,et al.  Games-Based Learning Advancements for Multi-Sensory Human Computer Interfaces: Techniques and Effective Practices , 2009 .

[28]  Herre van Oostendorp,et al.  Current Practices in Serious Game Research: A Review from a Learning Outcomes Perspective , 2009 .

[29]  Herre van Oostendorp,et al.  Attentional Cueing in Serious Games , 2010, 2010 Second International Conference on Games and Virtual Worlds for Serious Applications.

[30]  Timothy E. Goldsmith,et al.  Specificity of Structural Assessment of Knowledge , 2010 .

[31]  Herre van Oostendorp,et al.  Code Red: Triage, Or, COgnition-based DEsign Rules Enhancing Decisionmaking TRaining In A Game Environment , 2011 .