Artificial intelligence in health care: accountability and safety

Abstract The prospect of patient harm caused by the decisions made by an artificial intelligence-based clinical tool is something to which current practices of accountability and safety worldwide have not yet adjusted. We focus on two aspects of clinical artificial intelligence used for decision-making: moral accountability for harm to patients; and safety assurance to protect patients against such harm. Artificial intelligence-based tools are challenging the standard clinical practices of assigning blame and assuring safety. Human clinicians and safety engineers have weaker control over the decisions reached by artificial intelligence systems and less knowledge and understanding of precisely how the artificial intelligence systems reach their decisions. We illustrate this analysis by applying it to an example of an artificial intelligence-based system developed for use in the treatment of sepsis. The paper ends with practical suggestions for ways forward to mitigate these concerns. We argue for a need to include artificial intelligence developers and systems safety engineers in our assessments of moral accountability for patient harm. Meanwhile, none of the actors in the model robustly fulfil the traditional conditions of moral accountability for the decisions of an artificial intelligence system. We should therefore update our conceptions of moral accountability in this context. We also need to move from a static to a dynamic model of assurance, accepting that considerations of safety are not fully resolvable during the design of the artificial intelligence system before the system has been deployed.

[1]  Christopher Rowe,et al.  Aristotle: Nicomachean Ethics , 2004 .

[2]  Geraint Rees,et al.  Clinically applicable deep learning for diagnosis and referral in retinal disease , 2018, Nature Medicine.

[3]  Richard Hawkins,et al.  A Pattern for Arguing the Assurance of Machine Learning in Medical Diagnosis Systems , 2019, SAFECOMP.

[4]  Aldo A. Faisal,et al.  The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care , 2018, Nature Medicine.

[5]  Dean F Sittig,et al.  A new sociotechnical model for studying health information technology in complex adaptive healthcare systems , 2010, Quality and Safety in Health Care.

[6]  J. Denny,et al.  Artificial intelligence, bias and clinical safety , 2019, BMJ Quality & Safety.

[7]  V. I. Cheadle,et al.  AN EVALUATION OF STUDIES ON ULTRASTRUCTURE OF TONOPLAST IN SIEVE ELEMENTS. , 1962, Proceedings of the National Academy of Sciences of the United States of America.

[8]  I. Habli,et al.  Human factors challenges for the safe use of artificial intelligence in patient care , 2019, BMJ Health & Care Informatics.

[9]  R. Wachter.  Personal accountability in healthcare: searching for the right balance , 2012, BMJ quality & safety.

[10]  Ewen Denney,et al.  Dynamic Safety Cases for Through-Life Safety Assurance , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[11]  F. V. van Haren,et al.  Fluid resuscitation in human sepsis: Time to rewrite history? , 2017, Annals of Intensive Care.

[12]  Raul Hakli,et al.  Moral Responsibility of Robots and Hybrid Agents , 2019, The Monist.

[13]  R. Bellomo,et al.  The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). , 2016, JAMA.

[14]  Jonathan Riddell,et al.  Evidence : Using safety cases in industry and healthcare , 2012 .

[15]  Jin-Oh Hahn,et al.  Hypotension in ICU Patients Receiving Vasopressor Therapy , 2017, Scientific Reports.

[16]  John Schulman,et al.  Concrete Problems in AI Safety , 2016, ArXiv.

[17]  C. Sprung,et al.  Surviving Sepsis Campaign: International Guidelines for Management of Severe Sepsis and Septic Shock, 2012 , 2013, Intensive Care Medicine.

[18]  Donald M Yealy,et al.  Severe sepsis in pre-hospital emergency care: analysis of incidence, care, and outcome. , 2012, American journal of respiratory and critical care medicine.

[19]  M. Talbert.  Moral Responsibility: An Introduction , 2016 .

[20]  M. Dixon-Woods,et al.  What is the role of individual accountability in patient safety? A multi‐site ethnographic study , 2015, Sociology of health & illness.

[21]  John Martin Fischer,et al.  Responsibility and Control: A Theory of Moral Responsibility , 1998 .

[22]  P. Strawson.  Freedom and Resentment , 1962 .

[23]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[24]  Aaron S Kesselheim,et al.  Lifecycle Regulation of Artificial Intelligence- and Machine Learning-Based Software Devices in Medicine. , 2019, JAMA.

[25]  E. Shortliffe,et al.  Clinical Decision Support in the Era of Artificial Intelligence. , 2018, JAMA.