A Model-Based Measure to Assess Operator Adherence to Procedures

Procedures play an important role in domains where humans interact with critical, complex systems. In such environments, the operator’s ability to correctly follow a given set of procedures can directly impact system safety. A quantitative measure of procedural adherence during training for complex system operation would be useful to assess trainee performance and evaluate a training program. This paper presents a novel model-based objective metric for quantifying procedural adherence in training. This metric is sensitive to both the number and nature of procedural deviations, and can be used with cluster analysis to classify trainee performance based on adherence. The metric was tested on an experimental data set gathered from volunteers using aircraft maintenance computer-based training (CBT). The properties of the metric are discussed, along with future possibilities.

[1]  John R. Anderson,et al.  How people learn to skip steps. , 1996 .

[2]  Cara J Hamann,et al.  Teaching and assessing procedural skills using simulation: metrics and methodology. , 2008, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[3]  Luiz Faillace Farret.  Using Virtual Reality Technology For Aircraft Visual Inspection Training: Presence and Comparison Studies. , 2012 .

[4]  George T. Frampton,et al.  Three Mile Island: a report to the commissioners and to the public. Volume II, Part 1 , 1979 .

[5]  Robert A. Bjork,et al.  Evaluating training during training: Obstacles and opportunities. , 1997 .

[6]  Harry Owen,et al.  Comparison of three simulation-based training methods for management of medical emergencies. , 2006, Resuscitation.

[7]  Eugene W. Myers,et al.  Suffix arrays: a new method for on-line string searches , 1993, SODA '90.

[8]  Marjan J. B. Govaerts,et al.  Broadening Perspectives on Clinical Performance Assessment: Rethinking the Nature of In-training Assessment , 2007, Advances in health sciences education : theory and practice.

[9]  Bruno Arnaldi,et al.  GVT: a platform to create virtual environments for procedural training , 2008, 2008 IEEE Virtual Reality Conference.

[10]  Mitchell Rogovin,et al.  Three Mile Island : a report to the Commissioners and to the public , 1980 .

[11]  Emilio Parrado-Hernández,et al.  A New Distance Measure for Model-Based Sequence Clustering , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  W. Hamman.  The complexity of team training: what we have learned from aviation and its applications to medicine , 2004, Quality and Safety in Health Care.

[13]  Robert A. Bjork,et al.  Optimizing Treatment and Instruction: Implications of a New Theory of Disuse , 2013 .

[14]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .