Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead

Black box machine learning models are currently being used for high-stakes decision making throughout society, causing problems in healthcare, criminal justice and other domains. Some people hope that creating methods for explaining these black box models will alleviate some of the problems, but trying to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practice and can potentially cause great harm to society. The way forward is to design models that are inherently interpretable. This Perspective clarifies the chasm between explaining black boxes and using inherently interpretable models, outlines several key reasons why explainable black boxes should be avoided in high-stakes decisions, identifies challenges to interpretable machine learning, and provides several example applications where interpretable models could potentially replace black box models in criminal justice, healthcare and computer vision.There has been a recent rise of interest in developing methods for ‘explainable AI’, where models are created to explain how a first ‘black box’ machine learning model arrives at a specific decision. It can be argued that instead efforts should be directed at building inherently interpretable models in the first place, in particular where they are applied in applications that directly affect human lives, such as in healthcare and criminal justice.

[1]  Kush R. Varshney,et al.  Interpretable Two-level Boolean Rule Learning for Classification , 2015, ArXiv.

[2]  Cynthia Rudin,et al.  Modeling recovery curves with application to prostatectomy. , 2015, Biostatistics.

[3]  Cynthia Rudin,et al.  An optimization approach to learning falling rule lists , 2017, AISTATS.

[4]  Steve Oudot,et al.  Statistical Analysis and Parameter Selection for Mapper , 2017, J. Mach. Learn. Res..

[5]  Sanjeeb Dash,et al.  Boolean Decision Rules via Column Generation , 2018, NeurIPS.

[6]  Cynthia Rudin,et al.  A study in Rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning , 2019, ArXiv.

[7]  Seth Flaxman,et al.  EU regulations on algorithmic decision-making and a "right to explanation" , 2016, ArXiv.

[8]  Cynthia Rudin,et al.  The World Health Organization Adult Attention-Deficit/Hyperactivity Disorder Self-Report Screening Scale for DSM-5 , 2017, JAMA psychiatry.

[9]  Johannes Gehrke,et al.  Accurate intelligible models with pairwise interactions , 2013, KDD.

[10]  David J. Hand,et al.  Classifier Technology and the Illusion of Progress , 2006, math/0606441.

[11]  Cynthia Rudin,et al.  Supersparse linear integer models for optimized medical scoring systems , 2015, Machine Learning.

[12]  Robert C. Holte,et al.  Very Simple Classification Rules Perform Well on Most Commonly Used Datasets , 1993, Machine Learning.

[13]  G. A. Miller THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[14]  Patricia Neri,et al.  Introduction: 2017 Daniel H. Wagner Prize for Excellence in Operations Research Practice , 2018, Interfaces.

[15]  Cynthia Rudin,et al.  How to reverse-engineer quality rankings , 2012, Machine Learning.

[16]  Emilio Carrizosa,et al.  Binarized Support Vector Machines , 2010, INFORMS J. Comput..

[17]  Haimonti Dutta,et al.  A process for predicting manhole events in Manhattan , 2009, Machine Learning.

[18]  Ashok K. Goel Editorial: Expository AI Applications , 2017, AI Mag..

[19]  Cynthia Rudin,et al.  Deep Learning for Case-based Reasoning through Prototypes: A Neural Network that Explains its Predictions , 2017, AAAI.

[20]  Bart Baesens,et al.  An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models , 2011, Decis. Support Syst..

[21]  Lawrence Carin,et al.  Cross-Spectral Factor Analysis , 2017, NIPS.

[22]  Cynthia Rudin,et al.  Optimized Risk Scores , 2017, KDD.

[23]  Nataliya Sokolovska,et al.  A Provable Algorithm for Learning Interpretable Scoring Systems , 2018, AISTATS.

[24]  Cynthia Rudin,et al.  Falling Rule Lists , 2014, AISTATS.

[25]  Elizabeth Mannshardt,et al.  Air quality in the USA , 2018, Significance.

[26]  Cynthia Rudin,et al.  An Interpretable Model with Globally Consistent Explanations for Credit Risk , 2018, ArXiv.

[27]  Chris Russell,et al.  Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR , 2017, ArXiv.

[28]  Peter Auer,et al.  Theory and Applications of Agnostic PAC-Learning with Small Decision Trees , 1995, ICML.

[29]  Christopher T. Lowenkamp,et al.  False Positives, False Negatives, and False Analyses: A Rejoinder to "Machine Bias: There's Software Used across the Country to Predict Future Criminals. and It's Biased against Blacks" , 2016 .

[30]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[31]  Stefan Rüping,et al.  Learning interpretable models , 2006 .

[32]  Cynthia Rudin,et al.  Please Stop Explaining Black Box Models for High Stakes Decisions , 2018, ArXiv.

[33]  N. Tollenaar,et al.  Which method predicts recidivism best?: a comparison of statistical, machine learning and data mining predictive models , 2013 .

[34]  Cynthia Rudin,et al.  Optimized Scoring Systems: Toward Trust in Machine Learning for Healthcare and Criminal Justice , 2018, Interfaces.

[35]  Maya R. Gupta,et al.  Monotonic Calibrated Interpolated Look-Up Tables , 2015, J. Mach. Learn. Res..

[36]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[37]  Cynthia Rudin,et al.  Box drawings for learning with imbalanced data , 2014, KDD.

[38]  François Laviolette,et al.  Domain-Adversarial Training of Neural Networks , 2015, J. Mach. Learn. Res..

[39]  Cynthia Rudin,et al.  The age of secrecy and unfairness in recidivism prediction , 2018, 2.1.

[40]  Cynthia Rudin,et al.  A Bayesian Framework for Learning Rule Sets for Interpretable Classification , 2017, J. Mach. Learn. Res..

[41]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[42]  Jenna Wiens,et al.  Learning Credible Models , 2018, KDD.

[43]  Chris Russell,et al.  Explaining Explanations in AI , 2018, FAT.

[44]  Alex Alves Freitas,et al.  Comprehensible classification models: a position paper , 2014, SKDD.

[45]  Padhraic Smyth,et al.  From Data Mining to Knowledge Discovery in Databases , 1996, AI Mag..

[46]  Johannes Gehrke,et al.  Intelligible models for classification and regression , 2012, KDD.

[47]  David A. Sontag,et al.  Population-Level Prediction of Type 2 Diabetes From Claims Data and Analysis of Risk Factors , 2015, Big Data.

[48]  Kush R. Varshney,et al.  On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products , 2016, Big Data.

[49]  Cynthia Rudin,et al.  This Looks Like That: Deep Learning for Interpretable Image Recognition , 2018 .

[50]  Jan A. Kors,et al.  Finding a short and accurate decision rule in disjunctive normal form by exhaustive search , 2010, Machine Learning.

[51]  Cynthia Rudin,et al.  Interpretable classification models for recidivism prediction , 2015, 1503.07810.

[52]  N. Cowan,et al.  The Magical Mystery Four , 2010, Current directions in psychological science.

[53]  Seth Flaxman,et al.  European Union Regulations on Algorithmic Decision-Making and a "Right to Explanation" , 2016, AI Mag..

[54]  Timothy C. Au Random Forests, Decision Trees, and Categorical Predictors: The "Absent Levels" Problem , 2017, J. Mach. Learn. Res..

[55]  T. Brennan,et al.  Evaluating the Predictive Validity of the Compas Risk and Needs Assessment System , 2009 .

[56]  Luciano Floridi,et al.  Why a Right to Explanation of Automated Decision-Making Does Not Exist in the General Data Protection Regulation , 2017 .

[57]  Marcus A. Badgeley,et al.  Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study , 2018, PLoS medicine.