Seeing the Forest Through the Trees

Ensemble methods are popular learning methods that are usually able to increase the predictive accuracy of a classifier. On the other hand, this comes at the cost of interpretability, and insight in the decision process of an ensemble is hard to obtain. This is a major reason why ensemble methods have not been extensively used in the setting of inductive logic programming. In this paper we aim to overcome this issue of comprehensibility by learning a single first order interpretable model that approximates the first order ensemble. The new model is obtained by exploiting the class distributions predicted by the ensemble. These are employed to compute heuristics for deciding which tests are to be used in the new model. As such we obtain a model that is able to give insight in the decision process of the ensemble, while being more accurate than the single model directly learned on the data.

[1]  Ryszard S. Michalski,et al.  Pattern Recognition as Rule-Guided Inductive Inference , 1980, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[3]  Saso Dzeroski,et al.  First order random forests: Learning relational classifiers with complex aggregates , 2006, Machine Learning.

[4]  José Hernández-Orallo,et al.  From Ensemble Methods to Comprehensible Models , 2002, Discovery Science.

[5]  Susanne Hoche,et al.  Relational Learning Using Constrained Confidence-Rated Boosting , 2001, ILP.

[6]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[7]  David Page,et al.  An Empirical Evaluation of Bagging in Inductive Logic Programming , 2002, ILP.

[8]  Hendrik Blockeel,et al.  Seeing the Forest Through the Trees: Learning a Comprehensible Model from an Ensemble , 2007, ECML.

[9]  Mark Craven,et al.  Extracting comprehensible models from trained neural networks , 1996 .

[10]  Eric Bauer,et al.  An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants , 1999, Machine Learning.

[11]  Ashwin Srinivasan,et al.  Carcinogenesis Predictions Using ILP , 1997, ILP.

[12]  Bart Demoen,et al.  Improving the Efficiency of Inductive Logic Programming Through the Use of Query Packs , 2011, J. Artif. Intell. Res..

[13]  Zhi-Hua Zhou,et al.  Extracting symbolic rules from trained neural network ensembles , 2003, AI Commun..

[14]  Jan Ramon,et al.  Learning an interpretable model from an ensemble in ILP , 2006 .

[15]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[16]  J. Ross Quinlan,et al.  Boosting First-Order Learning , 1996, ALT.

[17]  金田 重郎,et al.  C4.5: Programs for Machine Learning (書評) , 1995 .

[18]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[19]  Hendrik Blockeel,et al.  Top-Down Induction of First Order Logical Decision Trees , 1998, AI Commun..

[20]  Pedro M. Domingos.  Knowledge Discovery Via Multiple Models , 1998, Intell. Data Anal..

[21]  Thomas Richardson,et al.  Interpretable Boosted Naïve Bayes Classification , 1998, KDD.