Rule Extraction from Ensemble Methods Using Aggregated Decision Trees

Ensemble methods have become very well known for being powerful pattern recognition algorithms capable of achieving high accuracy. However, Ensemble methods produces learners that are not comprehensible or transferable thus making them unsuitable for tasks that require a rational justification for making a decision. Rule Extraction methods can resolve this limitation by extracting comprehensible rules from a trained ensembles of classifiers. In this paper, we present an algorithm called REEMTIC that uses a symbolic learning algorithm (Decision Tree) on each underlying classifier of the ensemble and combines them. Experiments and theoretical analysis show REEMTIC generates highly accurate rules that closely approximates the Ensemble Learned Model.

[1]  Aiko M. Hormann,et al.  Programs for Machine Learning. Part I , 1962, Inf. Control..

[2]  J. Friedman.  Special Invited Paper-Additive logistic regression: A statistical view of boosting , 2000 .

[3]  Lars Kai Hansen,et al.  Neural Network Ensembles , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Henrik Jacobsson,et al.  Rule Extraction from Recurrent Neural Networks: ATaxonomy and Review , 2005, Neural Computation.

[5]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[6]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[7]  Robert K. Brayton,et al.  Logic Minimization Algorithms for VLSI Synthesis , 1984, The Kluwer International Series in Engineering and Computer Science.

[8]  Michael J. Pazzani,et al.  A Knowledge-intensive Approach to Learning Relational Concepts , 1991, ML.

[9]  Ashish Darbari,et al.  Rule Extraction from Trained ANN: A Survey , 2000 .

[10]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[11]  Jude W. Shavlik,et al.  Using Sampling and Queries to Extract Rules from Trained Neural Networks , 1994, ICML.

[12]  Sebastian Thrun,et al.  Extracting Rules from Artifical Neural Networks with Distributed Representations , 1994, NIPS.

[13]  Joachim Diederich,et al.  Survey and critique of techniques for extracting rules from trained artificial neural networks , 1995, Knowl. Based Syst..

[14]  D. Opitz,et al.  Popular Ensemble Methods: An Empirical Study , 1999, J. Artif. Intell. Res..

[15]  Jude W. Shavlik,et al.  in Advances in Neural Information Processing , 1996 .

[16]  Sherif Hashem,et al.  Optimal Linear Combinations of Neural Networks , 1997, Neural Networks.

[17]  Richi Nayak,et al.  GYAN: A methodology for rule extraction from artificial neural networks , 1999 .