Dependency Bagging

In this paper, a new variant of Bagging named DepenBag is proposed. This algorithm obtains bootstrap samples at first. Then, it employs a causal discoverer to induce from each sample a dependency model expressed as a Directed Acyclic Graph (DAG). The attributes without connections to the class attribute in all the DAGs are then removed. Finally, a component learner is trained from each of the resulted samples to constitute the ensemble. Empirical study shows that DepenBag is effective in building ensembles of nearest neighbor classifiers.

[1]  Gang Li,et al.  An Improved Approach for the Discovery of Causal Models via MML , 2002, PAKDD.

[2]  Kevin Murphy,et al.  Bayes net toolbox for Matlab , 1999 .

[3]  Michael A. Arbib,et al.  The handbook of brain theory and neural networks , 1995, A Bradford book.

[4]  Gregory F. Cooper,et al.  A Bayesian method for the induction of probabilistic networks from data , 1992, Machine Learning.

[5]  ChickeringDavid Maxwell.  Learning equivalence classes of bayesian-network structures , 2002 .

[6]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[7]  Lawrence O. Hall,et al.  Comparing pure parallel ensemble creation techniques against bagging , 2003, Third IEEE International Conference on Data Mining.

[8]  Josef Kittler,et al.  Moderating k-NN Classifiers , 2002, Pattern Analysis & Applications.

[9]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[10]  David Maxwell Chickering,et al.  Learning Bayesian Networks: The Combination of Knowledge and Statistical Data , 1994, Machine Learning.

[11]  David W. Aha,et al.  Special Issue on Lazy Learning , 1997 .

[12]  Ludmila I. Kuncheva.  Diversity in multiple classifier systems , 2005, Inf. Fusion.

[13]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[14]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[15]  Belur V. Dasarathy,et al.  Nearest neighbor (NN) norms: NN pattern classification techniques , 1991 .

[16]  R. W. Robinson.  Counting unlabeled acyclic digraphs , 1977 .

[17]  Igor Kononenko,et al.  Estimating Attributes: Analysis and Extensions of RELIEF , 1994, ECML.

[18]  Huan Liu,et al.  Chi2: feature selection and discretization of numeric attributes , 1995, Proceedings of 7th IEEE International Conference on Tools with Artificial Intelligence.

[19]  Nikunj C. Oza,et al.  Online Ensemble Learning , 2000, AAAI/IAAI.

[20]  Richard E. Neapolitan,et al.  Learning Bayesian networks , 2007, KDD '07.

[21]  Olivier Debeir,et al.  Mixing Bagging and Multiple Feature Subsets to Improve Classification Accuracy of Decision Tree Combination , 2000 .

[22]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[23]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[24]  Yang Yu,et al.  Adapt Bagging to Nearest Neighbor Classifiers , 2005, Journal of Computer Science and Technology.