Explainability for artificial intelligence in healthcare: a multidisciplinary perspective

Background Explainability is one of the most heavily debated topics when it comes to the application of artificial intelligence (AI) in healthcare. Even though AI-driven systems have been shown to outperform humans in certain analytical tasks, the lack of explainability continues to spark criticism. Yet, explainability is not a purely technological issue, instead it invokes a host of medical, legal, ethical, and societal questions that require thorough exploration. This paper provides a comprehensive assessment of the role of explainability in medical AI and makes an ethical evaluation of what explainability means for the adoption of AI-driven tools into clinical practice. Methods Taking AI-based clinical decision support systems as a case in point, we adopted a multidisciplinary approach to analyze the relevance of explainability for medical AI from the technological, legal, medical, and patient perspectives. Drawing on the findings of this conceptual analysis, we then conducted an ethical assessment using the “Principles of Biomedical Ethics” by Beauchamp and Childress (autonomy, beneficence, nonmaleficence, and justice) as an analytical framework to determine the need for explainability in medical AI. Results Each of the domains highlights a different set of core considerations and values that are relevant for understanding the role of explainability in clinical practice. From the technological point of view, explainability has to be considered both in terms how it can be achieved and what is beneficial from a development perspective. When looking at the legal perspective we identified informed consent, certification and approval as medical devices, and liability as core touchpoints for explainability. Both the medical and patient perspectives emphasize the importance of considering the interplay between human actors and medical AI. We conclude that omitting explainability in clinical decision support systems poses a threat to core ethical values in medicine and may have detrimental consequences for individual and public health. Conclusions To ensure that medical AI lives up to its promises, there is a need to sensitize developers, healthcare professionals, and legislators to the challenges and limitations of opaque algorithms in medical AI and to foster multidisciplinary collaboration moving forward.

[1]  Alastair Baker,et al.  Crossing the Quality Chasm: A New Health System for the 21st Century , 2001, BMJ : British Medical Journal.

[2]  Ioannis A. Kakadiaris,et al.  Machine Learning Outperforms ACC/AHA CVD Risk Calculator in MESA , 2018, Journal of the American Heart Association.

[3]  R. McDougall.  Computer knows best? The need for value-flexibility in medical AI , 2018, Journal of Medical Ethics.

[4]  Hilde van der Togt,et al.  Publisher's Note , 2003, J. Netw. Comput. Appl..

[5]  Derek Doran,et al.  What Does Explainable AI Really Mean? A New Conceptualization of Perspectives , 2017, CEx@AI*IA.

[6]  Victor M Montori,et al.  What Is Shared Decision Making? (and What It Is Not). , 2016, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[7]  Jens Christian Bjerring,et al.  Artificial Intelligence and Patient-Centered Decision-Making , 2020, Philosophy & Technology.

[8]  Wojciech Samek,et al.  Toward Interpretable Machine Learning: Transparent Deep Neural Networks and Beyond , 2020, ArXiv.

[9]  Mary C Politi,et al.  Importance of clarifying patients’ desired role in shared decision making to match their level of engagement with their preferences , 2013, BMJ.

[10]  Maxine Mackintosh,et al.  Machine intelligence in healthcare—perspectives on trustworthiness, explainability, usability, and transparency , 2020, npj Digital Medicine.

[11]  Anna Goldenberg,et al.  What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use , 2019, MLHC.

[12]  Alexander Binder,et al.  Unmasking Clever Hans predictors and assessing what machines really learn , 2019, Nature Communications.

[13]  Xiaoxi Yao,et al.  Subclinical and Device-Detected Atrial Fibrillation: Pondering the Knowledge Gap: A Scientific Statement From the American Heart Association. , 2019, Circulation.

[14]  Alex John London,et al.  Artificial Intelligence and Black-Box Medical Decisions: Accuracy versus Explainability. , 2019, The Hastings Center report.

[15]  Philipp Berens,et al.  On the ethics of algorithmic decision-making in healthcare , 2019, Journal of Medical Ethics.

[16]  Cynthia Rudin,et al.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead , 2018, Nature Machine Intelligence.

[17]  S. Athar.  Principles of Biomedical Ethics , 2011, The Journal of IMA.

[18]  Wojciech Samek,et al.  Explainable AI: Interpreting, Explaining and Visualizing Deep Learning , 2019, Explainable AI.

[19]  Agata Ferretti,et al.  Machine Learning in Medicine , 2018 .

[20]  William Eberle,et al.  Towards Quantification of Explainability in Explainable Artificial Intelligence Methods , 2019, FLAIRS.

[21]  M. Barry,et al.  Shared decision making--pinnacle of patient-centered care. , 2012, The New England journal of medicine.

[22]  Marcus A. Badgeley,et al.  Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study , 2018, PLoS medicine.

[23]  Brent Mittelstadt,et al.  Principles alone cannot guarantee ethical AI , 2019, Nature Machine Intelligence.

[24]  Raanan Gillon,et al.  Defending the four principles approach as a good basis for good medical practice and therefore for good medical ethics , 2014, Journal of Medical Ethics.

[25]  Glyn Elwyn,et al.  Availability of patient decision aids for stroke prevention in atrial fibrillation: A systematic review , 2017, American heart journal.

[26]  R. Thomson,et al.  Decision aids for people facing health treatment or screening decisions. , 2011, The Cochrane database of systematic reviews.

[27]  James R. Eagan,et al.  Identifying the 'Right' Level of Explanation in a Given Situation , 2020, NeHuAI@ECAI.

[28]  Tianyu Liu,et al.  A hybrid machine learning approach to cerebral stroke prediction based on imbalanced medical dataset , 2019, Artif. Intell. Medicine.

[29]  I. Glenn Cohen,et al.  Informed Consent and Medical Artificial Intelligence: What to Tell the Patient? , 2020 .

[30]  Peter Noseworthy,et al.  Normalization of a conversation tool to promote shared decision making about anticoagulation in patients with atrial fibrillation within a practical randomized trial of its effectiveness: a cross-sectional study , 2020, Trials.

[31]  Ralf Krestel,et al.  Explainable AI under contract and tort law: legal incentives and technical challenges , 2020, Artificial Intelligence and Law.

[32]  Julia Hörnle.  Juggling more than three balls at once: multilevel jurisdictional challenges in EU Data Protection Regulation , 2019, Int. J. Law Inf. Technol..

[33]  E. Shortliffe,et al.  Clinical Decision Support in the Era of Artificial Intelligence. , 2018, JAMA.

[34]  David Higgins,et al.  From Bit to Bedside: A Practical Framework for Artificial Intelligence Product Development in Healthcare , 2020, Adv. Intell. Syst..

[35]  Kirsten McCaffery,et al.  Should heart age calculators be used alongside absolute cardiovascular disease risk assessment? , 2018, BMC Cardiovascular Disorders.

[36]  Brian W. Powers,et al.  Dissecting racial bias in an algorithm used to manage the health of populations , 2019, Science.

[37]  Daniel Schönberger,et al.  Artificial intelligence in healthcare: a critical analysis of the legal and ethical implications , 2019, Int. J. Law Inf. Technol..

[38]  Richard Bellamy,et al.  The morality of freedom , 1988 .

[39]  J. Kai,et al.  Can machine-learning improve cardiovascular risk prediction using routine clinical data? , 2017, PloS one.

[40]  R. Thomson,et al.  Decision aids for people facing health treatment or screening decisions. , 2003, The Cochrane database of systematic reviews.

[41]  B. Kapp,et al.  A history and theory of informed consent , 1986 .

[42]  Thomas T. Hildebrandt,et al.  What's in the Box? The Legal Requirement of Explainability in Computationally Aided Decision-Making in Public Administration , 2019 .

[43]  P. V. van Heerden,et al.  Ethical considerations about artificial intelligence for prognostication in intensive care , 2019, Intensive Care Medicine Experimental.

[44]  Andre Esteva,et al.  A guide to deep learning in healthcare , 2019, Nature Medicine.

[45]  Mohammad Hassan Murad,et al.  Impact of decision aids used during clinical encounters on clinician outcomes and consultation length: a systematic review , 2018, BMJ Quality & Safety.

[46]  Wojciech Samek,et al.  Explainable ai – preface , 2019 .

[47]  Victor M. Montori,et al.  Shared decision-making in atrial fibrillation: navigating complex issues in partnership with the patient , 2018, Journal of Interventional Cardiac Electrophysiology.