Are accident analysis methods fit for purpose? Testing the criterion-referenced concurrent validity of AcciMap, STAMP-CAST and AcciNet

Abstract Systems-based methods such as AcciMap, STAMP-CAST and more recently AcciNet, are used to uncover the multifactorial cause of accidents within complex systems to support the development of new safety interventions. Despite this, there has been a lack of research that has formally examined the reliability and validity of systems-based accident analysis methods. This gap is addressed in this study. The criterion-referenced concurrent validity of AcciMap, STAMP-CAST and AcciNet was compared in a repeated measures design using Signal Detection Theory (SDT). A process of analytical fragmentation was adopted to understand how the individual phases of each method contributed to their overall validity. The results of the overall analyses indicate that the three methods achieved comparable and moderate results. It is concluded that state-of-the-art systems-based accident analysis methods did not produce satisfactory validity results in this study compared to a referent expert analysis. Further research needs to be conducted to demonstrate the efficacy of systems-based accident methods if they are to be used with confidence and work as expected.

[1]  Patrick Waterson,et al.  Systems thinking, the Swiss Cheese Model and accident analysis: a comparative systemic analysis of the Grayrigg train derailment using the ATSB, AcciMap and STAMP models. , 2014, Accident; analysis and prevention.

[2]  Guy H. Walker,et al.  Models and methods for collision analysis: A comparison study based on the Uber collision with a pedestrian , 2019, Safety Science.

[3]  Guy H. Walker,et al.  Human Factors Methods and Accident Analysis: Practical Guidance and Case Study Applications , 2017 .

[4]  Neville A Stanton,et al.  Translating concepts of complexity to the field of ergonomics , 2010, Ergonomics.

[5]  Jens Rasmussen,et al.  Risk management in a dynamic society: a modelling problem , 1997 .

[6]  J. Carroll,et al.  Moving Beyond Normal Accidents and High Reliability Organizations: A Systems Approach to Safety in Complex Systems , 2009 .

[7]  Paul M. Salmon.  Bridging the gap between ergonomics methods research and practice: Methodological Issues in Ergonomics Science Part II , 2016 .

[8]  Guy H. Walker,et al.  What do applications of systems thinking accident analysis methods tell us about accident causation? A systematic review of applications between 1990 and 2018 , 2019, Safety Science.

[9]  Craig W. Johnson,et al.  An investigation into the loss of the Brazilian space programme’s launch vehicle VLS-1 V03 , 2008 .

[10]  Anastácio Pinto Gonçalves Filho,et al.  Four studies, two methods, one accident – An examination of the reliability and validity of Accimap and STAMP for accident analysis , 2019, Safety Science.

[11]  P. Waterson,et al.  Improving accident analysis in construction – Development of a contributing factor classification framework and evaluation of its validity and reliability , 2021 .

[12]  John Annett,et al.  A note on the validity and reliability of ergonomics methods , 2002 .

[13]  Nancy G. Leveson,et al.  Applying systems thinking to analyze and learn from events , 2010 .

[14]  Guy H. Walker,et al.  Testing the Reliability and Validity of Net-HARMS: A New Systems-Based Risk Assessment Method in HFE , 2021 .

[15]  H. Kanis,et al.  Reliability and validity of findings in ergonomics research , 2014 .

[16]  Guy H. Walker,et al.  Systems Thinking in Practice , 2018 .

[17]  Seyed Bagher Mortazavi,et al.  Validity and consistency assessment of accident analysis methods in the petroleum industry , 2019, International journal of occupational safety and ergonomics : JOSE.

[18]  Nancy G. Leveson,et al.  A new accident model for engineering safer systems , 2004 .

[19]  C. Finch,et al.  Developing a contributing factor classification scheme for Rasmussen's AcciMap: Reliability and validity evaluation. , 2017, Applied ergonomics.

[20]  Richard Taylor.  Interpretation of the Correlation Coefficient: A Basic Review , 1990 .

[21]  Neville A Stanton,et al.  Fitting methods to paradigms: are ergonomics methods fit for systems thinking? , 2017, Ergonomics.

[22]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[23]  Claire Williams,et al.  Human factors and ergonomics methods in practice: three fundamental constraints , 2016 .

[24]  Nancy G. Leveson,et al.  Engineering a Safer World: Systems Thinking Applied to Safety , 2012 .

[25]  P. Waterson,et al.  Complexity theory in accident causation: using AcciMap to identify the systems thinking tenets in 11 catastrophes , 2020, Ergonomics.

[26]  Nikki S Olsen.  Reliability studies of incident coding systems in high hazard industries: A narrative review of study methodology. , 2013, Applied ergonomics.

[27]  Patrick Waterson,et al.  Systemic accident analysis: examining the gap between research and practice. , 2013, Accident; analysis and prevention.

[28]  Mark S. Young,et al.  What price ergonomics? , 1999, Nature.

[29]  P. Waterson,et al.  The big picture on accident causation: A review, synthesis and meta-analysis of AcciMap studies , 2020 .

[30]  H. W. Heinrich,et al.  Industrial Accident Prevention: a Scientific Approach , 1951 .

[31]  Neville A. Stanton,et al.  On the reliability and validity of, and training in, ergonomics methods: a challenge revisited , 2016 .

[32]  Paul M. Salmon,et al.  Systems-based accident analysis methods: A comparison of Accimap, HFACS, and STAMP , 2012 .

[33]  Paul M. Salmon,et al.  Identifying risks and emergent risks across sociotechnical systems: the NETworked hazard analysis and risk management system (NET-HARMS) , 2017 .

[34]  Erik Hollnagel,et al.  Barriers And Accident Prevention , 2004 .

[35]  P. Waterson,et al.  Testing the reliability and validity of risk assessment methods in Human Factors and Ergonomics , 2021, Ergonomics.

[36]  Daniel P. Jenkins,et al.  'Remixing Rasmussen': The evolution of Accimaps within systemic accident analysis. , 2017, Applied ergonomics.

[37]  Natassia Goode,et al.  Evaluation of construct and criterion-referenced validity of a systems-thinking based near miss reporting form , 2019, Ergonomics.

[38]  P. Waterson,et al.  Systems thinking-based risk assessment methods applied to sports performance: A comparison of STPA, EAST-BL, and Net-HARMS in the context of elite women's road cycling. , 2020, Applied ergonomics.

[39]  Emmanuel Manatakis,et al.  Towards an evaluation of accident investigation methods in terms of their alignment with accident causation models , 2009 .

[40]  Scott A. Shappell,et al.  APPLYING REASON: THE HUMAN FACTORS ANALYSIS AND CLASSIFICATION SYSTEM (HFACS) , 2001 .

[41]  P. Ray,et al.  Occupational injury and accident research: A comprehensive review , 2012 .

[42]  The Accident Network (AcciNet): A new accident analysis method for describing the interaction between normal performance and failure , 2020 .

[43]  H Kanis,et al.  Questioning validity in the area of ergonomics/human factors , 2000, Ergonomics.

[44]  Neville A Stanton,et al.  Giving ergonomics away? The application of ergonomics methods by novices. , 2003, Applied ergonomics.

[45]  D. Chicco,et al.  The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation , 2020, BMC Genomics.

[46]  Neville A. Stanton,et al.  Defining the methodological challenges and opportunities for an effective science of sociotechnical systems and safety , 2015, Ergonomics.

[47]  Neville A. Stanton,et al.  Commentary on the paper by Heimrich Kanis entitled ‘Reliability and validity of findings in ergonomics research’: where is the methodology in ergonomics methods? , 2014 .

[48]  I. Svedung,et al.  Graphic representation of accident scenarios: mapping system structure and the causation of accidents , 2002 .

[49]  Sven Ove Hansson,et al.  Learning from accidents : what more do we need to know? , 2010 .

[50]  E. Hollnagel.  FRAM: The Functional Resonance Analysis Method: Modelling Complex Socio-technical Systems , 2012 .

[51]  Guy H. Walker,et al.  Methodological issues in systems Human Factors and Ergonomics: Perspectives on the research–practice gap, reliability and validity, and prediction , 2020, Human Factors and Ergonomics in Manufacturing & Service Industries.

[52]  B. Matthews.  Comparison of the predicted and observed secondary structure of T4 phage lysozyme. , 1975, Biochimica et biophysica acta.