On Shapley Credit Allocation for Interpretability

We emphasize the importance of asking the right question when interpreting the decisions of a learning model. We discuss a natural extension of the theoretical machinery from Janzing et. al. 2020, which answers the question "Why did my model predict a person has cancer?" for answering a more involved question, "What caused my model to predict a person has cancer?" While the former quantifies the direct effects of variables on the model, the latter also accounts for indirect effects, thereby providing meaningful insights wherever human beings can reason in terms of cause and effect. We propose three broad categories for interpretations: observational, model-specific and causal each of which are significant in their own right. Furthermore, this paper quantifies feature relevance by weaving different natures of interpretations together with different measures as characteristic functions for Shapley symmetrization. Besides the widely used expected value of the model, we also discuss measures of statistical uncertainty and dispersion as informative candidates, and their merits in generating explanations for each data point, some of which are used in this context for the first time. These measures are not only useful for studying the influence of variables on the model output, but also on the predictive performance of the model, and for that we propose relevant characteristic functions that are also used for the first time.

[1]  Yunmei Chen,et al.  Cumulative residual entropy: a new measure of information , 2004, IEEE Transactions on Information Theory.

[2]  Monika Doll,et al.  Cumulative Paired φ-Entropy , 2016, Entropy.

[3]  Ankur Taly,et al.  Axiomatic Attribution for Deep Networks , 2017, ICML.

[4]  Suresh Venkatasubramanian,et al.  Problems with Shapley-value-based explanations as feature importance measures , 2020, ICML.

[5]  Erik Strumbelj,et al.  An Efficient Explanation of Individual Classifications using Game Theory , 2010, J. Mach. Learn. Res..

[6]  Le Song,et al.  L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data , 2018, ICLR.

[7]  Dominik Janzing,et al.  Feature relevance quantification in explainable AI: A causality problem , 2019, AISTATS.

[8]  Franco Turini,et al.  A Survey of Methods for Explaining Black Box Models , 2018, ACM Comput. Surv..

[9]  W. Marsden.  I and J , 2012 .

[10]  Zachary Chase Lipton.  The mythos of model interpretability , 2016, ACM Queue.

[11]  Scott Lundberg,et al.  Understanding Global Feature Contributions Through Additive Importance Measures , 2020, ArXiv.

[12]  Settimo Termini,et al.  A Definition of a Nonprobabilistic Entropy in the Setting of Fuzzy Sets Theory , 1972, Inf. Control..

[13]  Illtyd Trethowan.  Causality , 1938 .

[14]  Thierry Chonavel,et al.  Generalized Cumulative Residual Entropy for Distributions with Unrestricted Supports , 2008, J. Electr. Comput. Eng..

[15]  Ingo Klein,et al.  (Generalized) Maximum Cumulative Direct, Residual, and Paired Φ Entropy Approach , 2020, Entropy.

[16]  S. Sunoj,et al.  Quantile based entropy function , 2012 .

[17]  Majid Asadi,et al.  Some new results on the cumulative residual entropy , 2010 .

[18]  Mukund Sundararajan,et al.  The many Shapley values for model explanation , 2019, ICML.

[19]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[20]  Sisi Ma,et al.  Predictive and Causal Implications of using Shapley Value for Model Interpretation , 2020, CD@KDD.

[21]  A. Charnes,et al.  Extremal Principle Solutions of Games in Characteristic Function Form: Core, Chebychev and Shapley Value Generalizations , 1988 .

[22]  L. Shapley.  Notes on the n-Person Game — I: Characteristic-Point Solutions of the Four-Person Game , 1951 .

[23]  Robert K. Leik,et al.  A Measure of Ordinal Consensus , 1966 .

[24]  Dominik Janzing,et al.  Quantifying causal contribution via structure preserving interventions , 2020, ArXiv.

[25]  Scott Lundberg,et al.  A Unified Approach to Interpreting Model Predictions , 2017, NIPS.

[26]  Scott M. Lundberg,et al.  Consistent Individualized Feature Attribution for Tree Ensembles , 2018, ArXiv.

[27]  Baoding Liu,et al.  Uncertainty Theory - A Branch of Mathematics for Modeling Human Uncertainty , 2011, Studies in Computational Intelligence.

[28]  Avanti Shrikumar,et al.  Learning Important Features Through Propagating Activation Differences , 2017, ICML.

[29]  Ronald R. Yager,et al.  Dissonance - A Measure of Variability for Ordinal Random Variables , 2001, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[30]  Neil Genzlinger.  A. and Q , 2006 .