A comparative study of patient and staff safety evaluation using tree-based machine learning algorithms

Abstract Medical errors constitute a significant challenge harming patients and healthcare staff in complex and dynamic healthcare systems. Various organizational factors may contribute to such errors; however, limited research have addressed patient and staff safety issues simultaneously in the same study setting. To evaluate this, we conduct an exploratory analysis using two types of tree-based machine learning algorithms, random forests and gradient boosting, using the hospital-level aggregate staff experience survey data from UK hospitals. Based on staff views and priorities, the results from both algorithms suggest that “health and wellbeing” is the leading theme associated with the number of reported errors and near misses harming patient and staff safety. Specifically, “work-related stress” is the most important survey item associated with safety outcomes. With respect to prediction accuracy, both algorithms provide similar results with comparable values in error metrics. Based on the analytical results, healthcare risk managers and decision-makers can develop and implement policies and practices that address staff experience and prioritize resources effectively to improve patient and staff safety.

[1]  Abigail R. Wooldridge,et al.  Work system barriers and facilitators in inpatient care transitions of pediatric trauma patients. , 2020, Applied ergonomics.

[2]  Beatriz Fernández-Muñiz,et al.  Safety culture: analysis of the causal relationships between its key dimensions. , 2007, Journal of safety research.

[3]  Xinxia Liu,et al.  Safety climate, safety behavior, and worker injuries in the Chinese manufacturing industry , 2015 .

[4]  Gulsum Kubra Kaya,et al.  A framework to support risk assessment in hospitals , 2018, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[5]  Niraj L. Sehgal,et al.  AHRQ's Hospital Survey on Patient Safety Culture: Psychometric Analyses , 2009, Journal of patient safety.

[6]  Athanasios Tsanas,et al.  Accurate quantitative estimation of energy performance of residential buildings using statistical machine learning tools , 2012 .

[7]  Denis Larocque,et al.  Mixed-effects random forest for clustered data , 2014 .

[8]  Jack Mardekian,et al.  Using Random Forest Models to Identify Correlates of a Diabetic Peripheral Neuropathy Diagnosis from Electronic Health Record Data , 2017, Pain medicine.

[9]  A. Bagnasco,et al.  Patient safety culture: an Italian experience. , 2011, Journal of clinical nursing.

[10]  Nicola Paltrinieri,et al.  Learning about risk: Machine learning for risk assessment , 2019, Safety Science.

[11]  M. Makary,et al.  Medical error—the third leading cause of death in the US , 2016, British Medical Journal.

[12]  Sara Singer,et al.  Workforce perceptions of hospital safety culture: development and validation of the patient safety climate in healthcare organizations survey. , 2007, Health services research.

[13]  G. La Torre,et al.  How Much do Needlestick Injuries Cost? A Systematic Review of the Economic Evaluations of Needlestick and Sharps Injuries Among Healthcare Personnel , 2016, Infection Control & Hospital Epidemiology.

[14]  S. Leka,et al.  Work, Health, Safety and Well-Being: Current State of the Art , 2018, Managing Health, Safety and Well-Being.

[15]  P. Georgiadou,et al.  Restructuring seriously damages well-being of workers: The case of the restructuring programme in local administration in Greece , 2017 .

[16]  Abroon Qazi,et al.  Evaluation of patient safety culture using a random forest algorithm , 2020, Reliab. Eng. Syst. Saf..

[17]  Jingsi Zhang,et al.  Comparing machine learning algorithms in predicting thermal sensation using ASHRAE Comfort Database II , 2020, Energy and Buildings.

[18]  M. Jorge Cardoso,et al.  Risk of COVID-19 among front-line health-care workers and the general community: a prospective cohort study , 2020, The Lancet Public Health.

[19]  Sophie Lambert-Lacroix,et al.  Random forests for global sensitivity analysis: A selective review , 2021, Reliab. Eng. Syst. Saf..

[20]  J. Friedman Greedy function approximation: A gradient boosting machine. , 2001 .

[21]  J. Stephens,et al.  Clinical, economic, and humanistic burden of needlestick injuries in healthcare workers , 2017, Medical devices.

[22]  Hanguk Ryu,et al.  Predicting types of occupational accidents at construction sites in Korea using random forest model , 2019 .

[23]  L. Kohn,et al.  To Err Is Human : Building a Safer Health System , 2007 .

[24]  J. Friedman Stochastic gradient boosting , 2002 .

[25]  M. D. Cooper,et al.  Criterion-related validity of the cultural web when assessing safety culture , 2019, Safety Science.

[26]  Samer Ellahham,et al.  Application of Artificial Intelligence in the Health Care Safety Context: Opportunities and Challenges , 2020, American journal of medical quality : the official journal of the American College of Medical Quality.

[27]  Oluseun Omotola Aremu,et al.  A machine learning approach to circumventing the curse of dimensionality in discontinuous time series machine data , 2020, Reliab. Eng. Syst. Saf..

[28]  Simone Pozzi,et al.  Should healthcare providers do safety cases? Lessons from a cross-industry review of safety case practices , 2016 .

[29]  Milton D. Hakel,et al.  Attitudes toward Surveys: Development of a Measure and Its Relationship to Respondent Behavior , 2001 .

[30]  Stefano Nembrini,et al.  The revival of the Gini importance? , 2018, Bioinform..

[31]  Xinyu Liang,et al.  Optimization design of micro-piles in landslide safety protection based on machine learning , 2019, Safety Science.

[32]  Alyssa K. McGonagle,et al.  Management Commitment to Safety, Teamwork, and Hospital Worker Injuries. , 2016, Journal of hospital administration.

[33]  M. Austin,et al.  An Investigation of Healthcare Worker Perception of Their Workplace Safety and Incidence of Injury , 2020, Workplace health & safety.

[34]  Aurélien Géron,et al.  Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems , 2017 .

[35]  K. Louis Luangkesorn,et al.  Machine learning of fire hazard model simulations for use in probabilistic safety assessments at nuclear power plants , 2019, Reliab. Eng. Syst. Saf..

[36]  Jon C. Helton,et al.  Implementation and evaluation of nonparametric regression procedures for sensitivity analysis of computationally demanding models , 2009, Reliab. Eng. Syst. Saf..

[37]  M. Slade,et al.  Evaluation of safety climate and employee injury rates in healthcare , 2016, Occupational and Environmental Medicine.

[38]  Damaris Zurell,et al.  Collinearity: a review of methods to deal with it and a simulation study evaluating their performance , 2013 .

[39]  Cordula Wagner,et al.  The role of patient safety culture in the causation of unintended events in hospitals. , 2012, Journal of clinical nursing.

[40]  P. François,et al.  The contribution of open comments to understanding the results from the Hospital Survey on Patient Safety Culture (HSOPS): A qualitative study , 2018, PloS one.

[41]  Achim Zeileis,et al.  BMC Bioinformatics BioMed Central Methodology article Conditional variable importance for random forests , 2008 .

[42]  Joseph H. Saleh,et al.  Machine learning for helicopter accident analysis using supervised classification: Inference, prediction, and implications , 2020, Reliab. Eng. Syst. Saf..

[43]  Mark-Alexander Sujan,et al.  An organisation without a memory: A qualitative study of hospital staff perceptions on reporting and organisational learning for patient safety , 2015, Reliab. Eng. Syst. Saf..

[44]  S. Ziebland,et al.  Understanding how front-line staff use patient experience data for service improvement: an exploratory case study evaluation , 2020, Health Services and Delivery Research.

[45]  Yan-li Nie,et al.  Hospital survey on patient safety culture in China , 2013, BMC Health Services Research.

[46]  S. Bodur,et al.  Validity and reliability of Turkish version of "Hospital Survey on Patient Safety Culture" and perception of patient safety in public hospitals in Turkey , 2010, BMC health services research.

[47]  S. Altın,et al.  A COMPREHENSIVE COMPARISON OF MACHINE LEARNING ALGORITHMS ON DIAGNOSING ASTHMA DISEASE AND COPD , 2020 .

[48]  J. Freischlag,et al.  Burnout and Medical Errors Among American Surgeons , 2010, Annals of surgery.

[49]  Gulsum Kubra Kaya,et al.  Semi-quantitative application to the Functional Resonance Analysis Method for supporting safety management in a complex health-care process , 2020, Reliab. Eng. Syst. Saf..

[50]  Mark-Alexander Sujan,et al.  On the application of Human Reliability Analysis in healthcare: Opportunities and challenges , 2020, Reliab. Eng. Syst. Saf..

[51]  Ali Dag,et al.  A Bayesian Belief Network-based probabilistic mechanism to determine patient no-show risk categories , 2020 .

[52]  Elie Azar,et al.  Evaluation of tree-based ensemble learning algorithms for building energy performance estimation , 2018 .

[53]  Brian E. Smith,et al.  Integration of multiple methods in identifying patient safety risks , 2019, Safety Science.

[54]  Johan Hellings,et al.  Challenging patient safety culture: survey results. , 2007, International journal of health care quality assurance.

[55]  Timothy J. Vogus,et al.  The Safety Organizing Scale: Development and Validation of a Behavioral Measure of Safety Culture in Hospital Nursing Units , 2007, Medical care.

[56]  I. Gilchrist,et al.  Towards a decision support tool for intensive care discharge: machine learning algorithm development using electronic healthcare data from MIMIC-III and Bristol, UK , 2019, BMJ Open.

[57]  DelenDursun,et al.  Predicting overall survivability in comorbidity of cancers , 2015 .

[58]  Raymond T. Lee,et al.  Correlates of physician burnout across regions and specialties: a meta-analysis , 2013, Human Resources for Health.

[59]  B. Long,et al.  Suffering in Silence: Medical Error and its Impact on Health Care Providers. , 2018, The Journal of emergency medicine.

[60]  Measuring Patient Safety Culture in Riyadh's Hospitals: A Comparison between Public and Private Hospitals. , 2009, The Journal of the Egyptian Public Health Association.

[61]  Jian Zhou,et al.  Slope stability prediction for circular mode failure using gradient boosting machine approach based on an updated database of case histories , 2019, Safety Science.

[62]  M. Hutchinson,et al.  Hierarchy and medical error: Speaking up when witnessing an error , 2020 .

[63]  Paola Zuccolotto,et al.  Analysis and correction of bias in Total Decrease in Node Impurity measures for tree-based algorithms , 2010, Stat. Comput..

[64]  Danielle A. Tucker,et al.  Public Sector Organizational Failure: A Study of Collective Denial in the UK National Health Service , 2020, Journal of Business Ethics.

[65]  Amir Hassan Zadeh,et al.  Predicting overall survivability in comorbidity of cancers: A data mining approach , 2015, Decis. Support Syst..

[66]  Simone Pozzi,et al.  How can health care organisations make and justify decisions about risk reduction? Lessons from a cross-industry review and a health care stakeholder consensus development process , 2017, Reliab. Eng. Syst. Saf..

[67]  Abroon Qazi,et al.  Adoption of a Data‐Driven Bayesian Belief Network Investigating Organizational Factors that Influence Patient Safety , 2020, Risk analysis : an official publication of the Society for Risk Analysis.

[68]  D. R. Kouabénan,et al.  Safety climate, perceived risk, and involvement in safety management , 2015 .

[69]  Torsten B Neilands,et al.  The Safety Attitudes Questionnaire: psychometric properties, benchmarking data, and emerging research , 2006, BMC Health Services Research.

[70]  James R Ward,et al.  Design for patient safety: a systems-based risk identification framework , 2018, Ergonomics.

[71]  R. Abdelhai,et al.  Assessing Patient Safety Culture and Factors Affecting It among Health Care Providers at Cairo University Hospitals , 2012 .

[72]  Rhona Flin,et al.  Measuring safety culture in healthcare: A case for accurate diagnosis , 2007 .

[73]  Mark-Alexander Sujan,et al.  A novel tool for organisational learning and its impact on safety culture in a hospital dispensary , 2012, Reliab. Eng. Syst. Saf..

[74]  Matthew J. Eckelman,et al.  Predictive modeling for US commercial building energy use: A comparison of existing statistical and machine learning algorithms using CBECS microdata , 2018 .

[75]  L. Dyrbye,et al.  Physician Burnout, Well‐being, and Work Unit Safety Grades in Relationship to Reported Medical Errors , 2018, Mayo Clinic proceedings.

[76]  W. Ambrosius,et al.  Application of Random Forests Methods to Diabetic Retinopathy Classification Analyses , 2014, PloS one.

[77]  P. François,et al.  Hospital survey on patient safety culture (HSOPS): variability of scoring strategies , 2017, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[78]  The Domino Effect of Medical Errors , 2018, American journal of medical quality : the official journal of the American College of Medical Quality.

[79]  Mecit Can Emre Simsekler,et al.  The link between healthcare risk identification and patient safety culture. , 2019, International journal of health care quality assurance.

[80]  Kanako Seto,et al.  Development and applicability of Hospital Survey on Patient Safety Culture (HSOPS) in Japan , 2011, BMC health services research.

[81]  Alan J Card,et al.  Trust-level risk identification guidance in the NHS East of England. , 2015, The International journal of risk & safety in medicine.

[82]  G. Kaya,et al.  Evaluation of the formal risk assessment practice in hospitals in England , 2020 .

[83]  Frauke Kreuter,et al.  Tree-based machine learning methods for survey research , 2019 .

[84]  Olga Lyashevska,et al.  Predictive analytics with gradient boosting in clinical medicine. , 2019, Annals of translational medicine.

[85]  Junho Song,et al.  Accelerated Monte Carlo system reliability analysis through machine-learning-based surrogate models of network connectivity , 2017, Reliab. Eng. Syst. Saf..

[86]  Annalee Yassi,et al.  Patient safety--worker safety: building a culture of safety to improve healthcare worker and patient well-being. , 2005, Healthcare quarterly.

[87]  Børge Rokseth,et al.  Applications of machine learning methods for engineering risk assessment – A review , 2020, Safety Science.

[88]  P. Sousa,et al.  The patient safety culture: a systematic review by characteristics of Hospital Survey on Patient Safety Culture dimensions , 2018, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[89]  Claudio M. Rocco,et al.  A Reliability Model for Assessing Corporate Governance Using Machine Learning Techniques , 2019, Reliab. Eng. Syst. Saf..

[90]  Saeed Piri,et al.  Missing care: A framework to address the issue of frequent missing values;The case of a clinical decision support system for Parkinson's disease , 2020, Decis. Support Syst..

[91]  S. Saint,et al.  Evaluation of the association between Hospital Survey on Patient Safety Culture (HSOPS) measures and catheter-associated infections: results of two national collaboratives , 2016, BMJ Quality & Safety.

[92]  David Williamson Shaffer,et al.  Quantifying the qualitative with epistemic network analysis: A human factors case study of task-allocation communication in a primary care team , 2018, IISE transactions on healthcare systems engineering.