Explaining Black Boxes on Sequential Data using Weighted Automata

Understanding how a learned black box works is of crucial interest for the future of Machine Learning. In this paper, we pioneer the question of the global interpretability of learned black box models that assign numerical values to symbolic sequential data. To tackle that task, we propose a spectral algorithm for the extraction of weighted automata (WA) from such black boxes. This algorithm does not require the access to a dataset or to the inner representation of the black box: the inferred model can be obtained solely by querying the black box, feeding it with inputs and analyzing its outputs. Experiments using Recurrent Neural Networks (RNN) trained on a wide collection of 48 synthetic datasets and 2 real datasets show that the obtained approximation is of great quality.

[1]  Dean Alderucci.  A SPECTRAL ALGORITHM FOR LEARNING HIDDEN MARKOV MODELS THAT HAVE SILENT STATES , 2015 .

[2]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[3]  Chihiro Shibata,et al.  Predicting Sequential Data with LSTMs Augmented with Strictly 2-Piecewise Input Vectors , 2016, ICGI.

[4]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[5]  Petr Motlícek,et al.  Conversion of Recurrent Neural Network Language Models to Weighted Finite State Transducers for Automatic Speech Recognition , 2012, INTERSPEECH.

[6]  Ariadna Quattoni,et al.  Spectral learning of weighted automata , 2014, Machine Learning.

[7]  Alex Alves Freitas,et al.  Comprehensible classification models: a position paper , 2014, SKDD.

[8]  Been Kim,et al.  Towards A Rigorous Science of Interpretable Machine Learning , 2017, 1702.08608.

[9]  C. Lee Giles,et al.  Extraction of rules from discrete-time recurrent neural networks , 1996, Neural Networks.

[10]  Colin de la Higuera,et al.  PAutomaC: a probabilistic automata and hidden Markov models learning competition , 2013, Machine Learning.

[11]  Fabio A. González,et al.  A Deep Learning Architecture for Image Representation, Visual Interpretability and Automated Basal-Cell Carcinoma Cancer Detection , 2013, MICCAI.

[12]  Cees Witteveen,et al.  Learning Driving Behavior by Timed Syntactic Pattern Recognition , 2011, IJCAI.

[13]  Raphaël Bailly.  Méthodes spectrales pour l'inférence grammaticale probabiliste de langages stochastiques rationnels , 2011 .

[14]  François Denis,et al.  Absolute Convergence of Rational Series Is Semi-decidable , 2009, LATA.

[15]  M. Droste,et al.  Handbook of Weighted Automata , 2009 .

[16]  Alex Graves,et al.  Generating Sequences With Recurrent Neural Networks , 2013, ArXiv.

[17]  Ariadna Quattoni,et al.  A Maximum Matching Algorithm for Basis Selection in Spectral Learning , 2017, AISTATS.

[18]  Rich Caruana,et al.  Model compression , 2006, KDD '06.

[19]  Henrik Jacobsson,et al.  Rule Extraction from Recurrent Neural Networks: ATaxonomy and Review , 2005, Neural Computation.

[20]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[21]  Eran Yahav,et al.  Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples , 2017, ICML.

[22]  Liva Ralaivola,et al.  Grammatical inference as a principal component analysis problem , 2009, ICML '09.

[23]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[24]  C. Lee Giles,et al.  Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks , 1992, Neural Computation.

[25]  Rémi Eyraud,et al.  Scikit-SpLearn : a toolbox for the spectral learning of weighted automata compatible with scikit-learn , 2017 .

[26]  John McGee,et al.  Discretization of Time Series Data , 2005, J. Comput. Biol..

[27]  Arvind Satyanarayan,et al.  The Building Blocks of Interpretability , 2018 .

[28]  Ariadna Quattoni,et al.  Results of the Sequence PredIction ChallengE (SPiCe): a Competition on Learning the Next Symbol in a Sequence , 2016, ICGI.

[29]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[30]  Franco Turini,et al.  A Survey of Methods for Explaining Black Box Models , 2018, ACM Comput. Surv..

[31]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[32]  François Denis,et al.  Rational stochastic languages , 2006, ArXiv.

[33]  Joelle Pineau,et al.  Multitask Spectral Learning of Weighted Automata , 2017, NIPS.

[34]  Mehryar Mohri,et al.  Generalization bounds for learning weighted automata , 2018, Theor. Comput. Sci..

[35]  Jack W. Carlyle,et al.  Realizations by Stochastic Finite Automata , 1971, J. Comput. Syst. Sci..

[36]  C. Lee Giles,et al.  An Empirical Evaluation of Recurrent Neural Network Rule Extraction , 2017, ArXiv.

[37]  Zachary Chase Lipton.  The mythos of model interpretability , 2016, ACM Queue.