Assessment of Structural Knowledge as a Training Outcome in Process Control Environments

Objective: The objective of the present studies was to apply a novel method for structural knowledge assessment to process control to assess the potential of its measures as a training outcome. Background: Traditionally, knowledge is assessed by verbal achievement tests on the subject matter. However, traditional methods are regarded as limited in their ability to assess higher-order learning or understanding. Method: Two experiments (Experiment 1, N = 41; Experiment 2, N = 50) were conducted in which participants were given a 4-hr training session on a simulated process control task. At a later testing session, participants worked on the task for 70 min and completed knowledge tests on declarative, procedural, and structural knowledge. Structural knowledge was measured with the computer-based Association Structure Test (AST), which combines an association task and Pathfinder network on the basis of relatedness ratings. Results: In both studies, structural knowledge was significantly related to diagnostic performance, and evidence was found for internal consistency as well as convergent and predictive validity. Conclusion: Findings indicate that structural assessment with the AST shows promise as a training outcome in process control. Application: Potential applications of this research include the improvement of training design, delivery, and evaluation.

[1]  D. Gopher,et al.  Practice under changing priorities: An approach to the training of complex skills☆ , 1989 .

[2]  Timothy E. Goldsmith,et al.  A structural assessment of classroom learning , 1990 .

[3]  Kurt Kraiger,et al.  Linking training objectives to evaluation criteria. , 1997 .

[4]  Roger W. Schvaneveldt,et al.  Measuring the Structure of Expertise , 1985, Int. J. Man Mach. Stud..

[5]  Rainer H. Kluwe,et al.  Acquisition of knowledge in the control of a simulated technical system , 1997 .

[6]  Mark A. Davis,et al.  Evaluating Cognitive Training Outcomes: Validity and Utility of Structural Knowledge Assessment , 2003 .

[7]  Z. Dienes,et al.  A theory of implicit and explicit knowledge , 1999, Behavioral and Brain Sciences.

[8]  Herbert A. Simon,et al.  THE MIND'S EYE IN CHESS , 1988 .

[9]  Nancy J. Cooke,et al.  Toward an on-line knowledge assessment methodology: Building on the relationship between knowing and doing , 1996 .

[10]  E. Day,et al.  Knowledge structures and the acquisition of a complex skill. , 2001, The Journal of applied psychology.

[11]  T. Goldsmith,et al.  Assessing structural similarity of graphs , 1990 .

[12]  J. A. Landeweerd,et al.  Internal representation of a process, fault diagnosis and fault correction , 1979 .

[13]  Ryan Womack,et al.  Introduction to R , 2010, IASSIST.

[14]  Roger W. Schvaneveldt,et al.  Pathfinder associative networks: studies in knowledge organization , 1990 .

[15]  Annette Kluge,et al.  The effects of heuristic rule training on operator performance in a simulated process control environment , 2008, Ergonomics.

[16]  Helmut Felix Friedrich,et al.  Kognition und Wissensdiagnose : Die Entwicklung und empirische Überprüfung des computerunterstützten wissensdiagnostischen Instrumentariums "Netzwerk-Elaborierungs-Technik (NET)" , 1999 .

[17]  Jürgen S. Sauer,et al.  Comparative study of three training methods for enhancing process control performance: Emphasis shift training, situation awareness training, and drill and practice , 2010, Comput. Hum. Behav..

[18]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[19]  W. N. Venables,et al.  An introduction to R : notes on R: a programming environment for data analysis and graphics, version 1.9.1 , 2002 .

[20]  K. Stumpfe,et al.  [Psychosocial stress]. , 1976, ZFA. Zeitschrift fur Allgemeinmedizin.

[21]  Roger W. Schvaneveldt,et al.  Knowledge Acquisition in Distributed Mission Training , 2002 .

[22]  Elwyn Edwards,et al.  The Human Operator in Process Control , 1974 .

[23]  E. Salas,et al.  Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. , 1993 .

[24]  J. Thayer,et al.  The Effect of Brief Situational Awareness Training in a Police Shooting Simulator: An Experimental Study , 2006 .

[25]  Eduardo Salas,et al.  Measuring Knowledge Organization as a Method for Assessing Learning during Training , 1995, Hum. Factors.

[26]  J. Gower.  Some distance properties of latent root and vector methods used in multivariate analysis , 1966 .

[27]  Oliver T. Wolf,et al.  Psychosocial stress induces working memory impairments in an n-back paradigm , 2008, Psychoneuroendocrinology.

[28]  J Sauer,et al.  A conceptual framework for designing micro-worlds for complex work domains: a case study of the Cabin Air Management System. , 2000, Computers in human behavior.

[29]  William J. Gerace,et al.  Probing physics students’ conceptual knowledge structures through term association , 2002 .

[30]  John R. Anderson.  A spreading activation theory of memory. , 1983 .

[31]  Bertolt Meyer.  The effects of computer-elicited structural and group knowledge on complex problem solving performance , 2008 .

[32]  F. Schmidt,et al.  The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. , 1998 .

[33]  Christopher D. Wickens,et al.  An introduction to human factors engineering , 1997 .