Measures of Model Interpretability for Model Selection

The literature lacks definitions for quantitative measures of model interpretability for automatic model selection to achieve high accuracy and interpretability, hence we define inherent model interpretability. We extend the work of Lipton et al. and Liu et al. from qualitative and subjective concepts of model interpretability to objective criteria and quantitative measures. We also develop another new measure called simplicity of sensitivity and illustrate prior, initial and posterior measurement. Measures are tested and validated with some measures recommended for use. It is demonstrated that high accuracy and high interpretability are jointly achievable with little to no sacrifice in either.

[1]  Robert A. Greenes,et al.  Clinical Decision Support: The Road Ahead , 2006 .

[2]  L. Duponchel,et al.  Support vector machines (SVM) in near infrared (NIR) spectroscopy: Focus on parameters optimization and model interpretation , 2009 .

[3]  R. Hatala,et al.  Learning Curves in Health Professions Education , 2015, Academic medicine : journal of the Association of American Medical Colleges.

[4]  Elliott Sober,et al.  Parsimony and predictive equivalence , 1996 .

[5]  Dekang Lin,et al.  An Information-Theoretic Definition of Similarity , 1998, ICML.

[6]  I. Sobola,et al.  Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates , 2001 .

[7]  S. Lloyd,et al.  Measures of complexity: a nonexhaustive list , 2001 .

[8]  Randal S. Olson,et al.  PMLB: a large benchmark suite for machine learning evaluation and comparison , 2017, BioData Mining.

[9]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[10]  Agus Sudjianto,et al.  Relative Entropy Based Method for Probabilistic Sensitivity Analysis in Engineering Design , 2006 .

[11]  Johannes Gehrke,et al.  Intelligible models for classification and regression , 2012, KDD.

[12]  James McDermott,et al.  Diagnosing a disorder in a classification benchmark , 2016, Pattern Recognit. Lett..

[13]  Leo Breiman,et al.  Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author) , 2001 .

[14]  M. Kendall.  The treatment of ties in ranking problems. , 1945, Biometrika.

[15]  D. Freedman,et al.  On the histogram as a density estimator:L2 theory , 1981 .

[16]  Motoaki Kawanabe,et al.  How to Explain Individual Classification Decisions , 2009, J. Mach. Learn. Res..

[17]  D. W. Scott.  On optimal and data based histograms , 1979 .

[18]  Jesmin Nahar,et al.  Microarray data classification using automatic SVM kernel selection. , 2007, DNA and cell biology.

[19]  Paulo J. G. Lisboa,et al.  Interpretability in Machine Learning - Principles and Practice , 2013, WILF.

[20]  Kate Smith-Miles,et al.  On learning algorithm selection for classification , 2006, Appl. Soft Comput..

[21]  Nozha Boujemaa,et al.  Conditionally Positive Definite Kernels for SVM Based Image Recognition , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[22]  Udo Seiffert,et al.  Quantitative Measurements of model interpretability for the analysis of spectral data , 2013, 2013 IEEE Symposium on Computational Intelligence and Data Mining (CIDM).

[23]  Herbert A. Sturges,et al.  The Choice of a Class Interval , 1926 .

[24]  Duane Szafron,et al.  Visual Explanation of Evidence with Additive Classifiers , 2006, AAAI.

[25]  Rich Caruana,et al.  An empirical comparison of supervised learning algorithms , 2006, ICML.

[26]  J. Mercer.  Functions of Positive and Negative Type, and their Connection with the Theory of Integral Equations , 1909 .

[27]  David S. Wishart,et al.  Applications of Machine Learning in Cancer Prediction and Prognosis , 2006, Cancer informatics.

[28]  C. Tsallis.  Possible generalization of Boltzmann-Gibbs statistics , 1988 .

[29]  Barnabás Póczos,et al.  Undercomplete Blind Subspace Deconvolution , 2007, J. Mach. Learn. Res..

[30]  Barnabás Póczos,et al.  Separation theorem for independent subspace analysis and its consequences , 2012, Pattern Recognit..

[31]  A. Rényi.  On Measures of Entropy and Information , 1961 .

[32]  Seth Lloyd,et al.  Information measures, effective complexity, and total information , 1996 .

[33]  Alessandra Alaniz Macedo,et al.  Windowing improvements towards more comprehensible models , 2016, Knowl. Based Syst..

[34]  Jason Weston,et al.  A user's guide to support vector machines. , 2010, Methods in molecular biology.

[35]  T. Stijnen,et al.  Review: a gentle introduction to imputation of missing values. , 2006, Journal of clinical epidemiology.

[36]  James Zou,et al.  Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016) , 2016 .

[37]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[38]  S S Stevens,et al.  On the Theory of Scales of Measurement. , 1946, Science.

[39]  Been Kim,et al.  Towards A Rigorous Science of Interpretable Machine Learning , 2017, 1702.08608.

[40]  Carlos Guedes Soares,et al.  Global sensitivity analysis based on entropy , 2008 .

[41]  Eta S. Berner,et al.  Clinical Decision Support Systems , 1999, Health Informatics.

[42]  Raphaël Féraud,et al.  Contact personalization using a score understanding method , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[43]  Slobodan Ribarić,et al.  Introduction to Pattern Recognition , 1988 .

[44]  Bart Baesens,et al.  Building Acceptable Classification Models , 2010, Data Mining.

[45]  Joachim M. Buhmann,et al.  On Relevant Dimensions in Kernel Feature Spaces , 2008, J. Mach. Learn. Res..

[46]  Zachary Chase Lipton.  The mythos of model interpretability , 2016, ACM Queue.

[47]  Alexander Binder,et al.  Explaining nonlinear classification decisions with deep Taylor decomposition , 2015, Pattern Recognit..

[48]  Steve Selvin,et al.  Statistical Analysis of Epidemiologic Data , 1991 .

[49]  Paul W. Fieguth,et al.  A new Mercer sigmoid kernel for clinical data classification , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[50]  Tim Miller,et al.  Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences , 2017, ArXiv.