Algebraic aggregation of random forests: towards explainability and rapid evaluation

Random Forests are one of the most popular classifiers in machine learning. The larger they are, the more precise the outcome of their predictions. However, this comes at a cost: it is increasingly difficult to understand why a Random Forest made a specific choice, and its running time for classification grows linearly with the size (number of trees). In this paper, we propose a method to aggregate large Random Forests into a single, semantically equivalent decision diagram which has the following two effects: (1) minimal, sufficient explanations for Random Forest-based classifications can be obtained by means of a simple three step reduction, and (2) the running time is radically improved. In fact, our experiments on various popular datasets show speed-ups of several orders of magnitude, while, at the same time, also significantly reducing the size of the required data structure.

[1]  R. I. Bahar,et al.  Algebraic decision diagrams and their applications , 1993, Proceedings of 1993 International Conference on Computer Aided Design (ICCAD).

[2]  Tiziana Margaria,et al.  Language-Driven Engineering: From General-Purpose to Purpose-Specific Languages , 2019, Computing and Software Science.

[3]  Franco Turini,et al.  A Survey of Methods for Explaining Black Box Models , 2018, ACM Comput. Surv..

[4]  Michele Lombardi,et al.  Embedding Decision Trees and Random Forests in Constraint Programming , 2015, CPAIOR.

[5]  Tony R. Martinez,et al.  Reducing Decision Tree Ensemble Size Using Parallel Decision DAGs , 2009, Int. J. Artif. Intell. Tools.

[6]  Randal C. Burns,et al.  Forest Packing: Fast, Parallel Decision Forests , 2018, SDM.

[7]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[8]  Tsutomu Sasao,et al.  A Random Forest Using a Multi-valued Decision Diagram on an FPGA , 2017, 2017 IEEE 47th International Symposium on Multiple-Valued Logic (ISMVL).

[9]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[10]  Hillol Kargupta,et al.  A Fourier spectrum-based approach to represent decision trees for mining data streams in mobile environments , 2004, IEEE Transactions on Knowledge and Data Engineering.

[11]  Fabio Somenzi,et al.  Efficient manipulation of decision diagrams , 2001, International Journal on Software Tools for Technology Transfer.

[12]  Sebastian Nowozin,et al.  Decision Jungles: Compact and Rich Models for Classification , 2013, NIPS.

[13]  Randal E. Bryant,et al.  Graph-Based Algorithms for Boolean Function Manipulation , 1986, IEEE Transactions on Computers.

[14]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[15]  Hendrik Blockeel,et al.  Seeing the Forest Through the Trees: Learning a Comprehensible Model from an Ensemble , 2007, ECML.

[16]  Giles Hooker,et al.  Interpreting Models via Single Tree Approximation , 2016, 1610.09036.

[17]  Houtao Deng,et al.  Interpreting tree ensembles with inTrees , 2018, International Journal of Data Science and Analytics.

[18]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[19]  Johannes Gehrke,et al.  Intelligible models for classification and regression , 2012, KDD.

[20]  Satoshi Hara,et al.  Making Tree Ensembles Interpretable: A Bayesian Model Selection Approach , 2016, AISTATS.

[21]  Bernhard Steffen,et al.  Large Random Forests: Optimisation for Rapid Evaluation , 2019, ArXiv.

[22]  Tiziana Margaria,et al.  DSLs for Decision Services: A Tutorial Introduction to Language-Driven Engineering , 2018, ISoLA.

[23]  Saharon Rosset,et al.  Lossless Compression of Random Forests , 2019, Journal of Computer Science and Technology.

[24]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[25]  Bernhard Steffen,et al.  Towards Explainability in Machine Learning: The Formal Methods Way , 2020, IT Professional.

[26]  Robert R. Freimuth,et al.  A weighted random forests approach to improve predictive performance , 2013, Stat. Anal. Data Min..

[27]  Pedro M. Domingos.  Knowledge Discovery Via Multiple Models , 1998, Intell. Data Anal..