Controversy Rules - Discovering Regions Where Classifiers (Dis-)Agree Exceptionally

Finding regions for which there is higher controversy among different classifiers is insightful with regards to the domain and our models. Such evaluation can falsify assumptions, assert some, or also, bring to the attention unknown phenomena. The present work describes an algorithm, which is based on the Exceptional Model Mining framework, and enables that kind of investigations. We explore several public datasets and show the usefulness of this approach in classification tasks. We show in this paper a few interesting observations about those well explored datasets, some of which are general knowledge, and other that as far as we know, were not reported before.

[1]  Arno Knobbe,et al.  Exceptional Model Mining , 2008, ECML/PKDD.

[2]  Jure Leskovec,et al.  Interpretable Decision Sets: A Joint Framework for Description and Prediction , 2016, KDD.

[3]  Scott M. Lundberg,et al.  Consistent Individualized Feature Attribution for Tree Ensembles , 2018, ArXiv.

[4]  Peter Norvig,et al.  Artificial intelligence - a modern approach, 2nd Edition , 2003, Prentice Hall series in artificial intelligence.

[5]  Aiko M. Hormann,et al.  Programs for Machine Learning. Part I , 1962, Inf. Control..

[6]  Hongjun Lu,et al.  NeuroRule: A Connectionist Approach to Data Mining , 1995, VLDB.

[7]  Abhishek Das,et al.  Grad-CAM: Why did you say that? , 2016, ArXiv.

[8]  Suresh Venkatasubramanian,et al.  Auditing black-box models for indirect influence , 2016, Knowledge and Information Systems.

[9]  Samuel Ritter,et al.  Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study , 2017, ICML.

[10]  Lawrence B. Holder,et al.  Substructure Discovery Using Minimum Description Length and Background Knowledge , 1993, J. Artif. Intell. Res..

[11]  C.J.H. Mann,et al.  Handbook of Data Mining and Knowledge Discovery , 2004 .

[12]  Erik Strumbelj,et al.  Quality of classification explanations with PRBF , 2012, Neurocomputing.

[13]  N. Altman.  An Introduction to Kernel and Nearest-Neighbor Nonparametric Regression , 1992 .

[14]  Yarin Gal,et al.  Uncertainty in Deep Learning , 2016 .

[15]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[16]  Wouter Duivesteijn,et al.  Understanding Where Your Classifier Does (Not) Work -- The SCaPE Model Class for EMM , 2014, 2014 IEEE International Conference on Data Mining.

[17]  Satoshi Hara,et al.  Making Tree Ensembles Interpretable , 2016, 1606.05390.

[18]  Panagiotis Papapetrou,et al.  A peek into the black box: exploring classifiers by randomization , 2014, Data Mining and Knowledge Discovery.

[19]  Marko Robnik-Sikonja,et al.  Explaining Classifications For Individual Instances , 2008, IEEE Transactions on Knowledge and Data Engineering.

[20]  Panagiotis Papapetrou,et al.  GoldenEye++: A Closer Look into the Black Box , 2015, SLDS.