Using bayesian networks for selecting classifiers in GP ensembles

In this paper we present a novel approach for combining GP-based ensembles by means of a Bayesian Network. The proposed system is able to effectively learn decision tree ensembles using two different strategies: decision trees ensembles are learned by means of boosted GP algorithm; the responses of the learned ensembles are combined using a Bayesian network, which also implements a selection strategy that reduces the size of the built ensembles.

[1]  Giandomenico Spezzano,et al.  A Cellular Genetic Programming Approach to Classification , 1999, GECCO.

[2]  William Nick Street,et al.  Ensemble Pruning Via Semi-definite Programming , 2006, J. Mach. Learn. Res..

[3]  Gregory F. Cooper,et al.  A Bayesian Method for Constructing Bayesian Belief Networks from Databases , 1991, UAI.

[4]  Fabio Roli,et al.  An approach to the automatic design of multiple classifier systems , 2001, Pattern Recognit. Lett..

[5]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[6]  C. N. Liu,et al.  Approximating discrete probability distributions with dependence trees , 1968, IEEE Trans. Inf. Theory.

[7]  Claudio De Stefano,et al.  Classifier Combination by Bayesian Networks for Handwriting Recognition , 2009, Int. J. Pattern Recognit. Artif. Intell..

[8]  Salvatore J. Stolfo,et al.  Cost Complexity-Based Pruning of Ensemble Classifiers , 2001, Knowledge and Information Systems.

[9]  Anikó Ekárt,et al.  Maintaining the Diversity of Genetic Programs , 2002, EuroGP.

[10]  Alberto Maria Segre,et al.  Programs for Machine Learning , 1994 .

[11]  Robert P. W. Duin,et al.  An experimental study on diversity for bagging and boosting with linear classifiers , 2002, Inf. Fusion.

[12]  Lawrence O. Hall,et al.  Ensemble diversity measures and their application to thinning , 2004, Inf. Fusion.

[13]  Lefteris Angelis,et al.  Selective fusion of heterogeneous classifiers , 2005, Intell. Data Anal..

[14]  Daniel Hernández-Lobato,et al.  An Analysis of Ensemble Pruning Techniques Based on Ordered Aggregation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Fabio Roli,et al.  Multiple classifier systems for robust classifier design in adversarial environments , 2010, Int. J. Mach. Learn. Cybern..

[16]  Christian Gagné,et al.  Ensemble learning for free with evolutionary algorithms? , 2007, GECCO '07.

[17]  Xin Yao,et al.  Simultaneous training of negatively correlated neural networks in an ensemble , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[18]  Giandomenico Spezzano,et al.  GP ensembles for large-scale data classification , 2006, IEEE Transactions on Evolutionary Computation.

[19]  Tom Burr,et al.  Causation, Prediction, and Search , 2003, Technometrics.

[20]  Xizhao Wang,et al.  Dynamic ensemble extreme learning machine based on sample entropy , 2012, Soft Comput..

[21]  Wei Fan,et al.  Bagging , 2009, Encyclopedia of Machine Learning.

[22]  Sean N. Raymond,et al.  White Dwarf - M Dwarf Pairs in the Sloan Digital Sky Survey (SDSS) , 2002 .

[23]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Giandomenico Spezzano,et al.  Training Distributed GP Ensemble With a Selective Algorithm Based on Clustering and Pruning for Pattern Classification , 2008, IEEE Transactions on Evolutionary Computation.

[25]  David Maxwell Chickering,et al.  Learning Bayesian Networks: The Combination of Knowledge and Statistical Data , 1994, Machine Learning.

[26]  Jean-Philippe Thiran,et al.  Information Theoretic Combination of Classifiers with Application to AdaBoost , 2007, MCS.

[27]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[28]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..

[29]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[30]  Ioannis T. Christou,et al.  A classifier ensemble approach to the TV-viewer profile adaptation problem , 2012, International Journal of Machine Learning and Cybernetics.

[31]  H. Iba.  Bagging, Boosting, and bloating in Genetic Programming , 1999 .

[32]  Daniel Hernández-Lobato,et al.  Pruning Adaptive Boosting Ensembles by Means of a Genetic Algorithm , 2006, IDEAL.

[33]  Thomas G. Dietterich,et al.  Pruning Adaptive Boosting , 1997, ICML.

[34]  Rakesh Agrawal,et al.  SPRINT: A Scalable Parallel Classifier for Data Mining , 1996, VLDB.

[35]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[36]  Chandrika Kamath,et al.  Inducing oblique decision trees with evolutionary algorithms , 2003, IEEE Trans. Evol. Comput..

[37]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[38]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[39]  Wei Tang,et al.  Selective Ensemble of Decision Trees , 2003, RSFDGrC.

[40]  Claudio De Stefano,et al.  A Hybrid Evolutionary Algorithm for Bayesian Networks Learning: An Application to Classifier Combination , 2010, EvoApplications.

[41]  David Heckerman,et al.  A Tutorial on Learning with Bayesian Networks , 1999, Innovations in Bayesian Networks.

[42]  Li Zhang,et al.  Cascaded cluster ensembles , 2012, Int. J. Mach. Learn. Cybern..

[43]  Nir Friedman,et al.  Bayesian Network Classifiers , 1997, Machine Learning.

[44]  Xin Yao,et al.  Evolutionary ensembles with negative correlation learning , 2000, IEEE Trans. Evol. Comput..