LocalBoost: A Parallelizable Approach to Boosting Classifiers

Ensemble learning is an active field of research with applications to a broad range of problems. Adaboost is a widely used ensemble approach, however, its computational burden is high because it uses an explicit diversity method for building the individual learners. To address this issue, we present a variant of Adaboost where the learners can be trained in parallel, exchanging information on a sparse collaborative communication that restricts the visibility among them. Experiments on 12 UCI datasets show that this approach is competitive in terms of generalization error but more efficient than Adaboost and two other parallel approximations of this algorithm.

[1]  Indranil Palit,et al.  Scalable and Parallel Boosting with MapReduce , 2012, IEEE Transactions on Knowledge and Data Engineering.

[2]  Xin Yao,et al.  An analysis of diversity measures , 2006, Machine Learning.

[3]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[4]  Cynthia Rudin,et al.  Precise Statements of Convergence for AdaBoost and arc-gv , 2007 .

[5]  Robi Polikar.  Ensemble learning , 2009, Scholarpedia.

[6]  Stéphane Vialle,et al.  Implementation of the AdaBoost Algorithm for Large Scale Distributed Environments: Comparing JavaSpace and MPJ , 2009, 2009 15th International Conference on Parallel and Distributed Systems.

[7]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[8]  Yuhua Tang,et al.  Parallization of Adaboost Algorithm through Hybrid MPI/OpenMP and Transactional Memory , 2011, 2011 19th International Euromicro Conference on Parallel, Distributed and Network-Based Processing.

[9]  Yunfeng Wu,et al.  Fusing output information in neural networks: ensemble performs better , 2003, Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE Cat. No.03CH37439).

[10]  Torsten Hoefler,et al.  Generic topology mapping strategies for large-scale parallel architectures , 2011, ICS '11.

[11]  Leo Breiman,et al.  Using Iterated Bagging to Debias Regressions , 2001, Machine Learning.

[12]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[13]  Peter Tiño,et al.  Managing Diversity in Regression Ensembles , 2005, J. Mach. Learn. Res..

[14]  Ponnuthurai N. Suganthan,et al.  Oblique Decision Tree Ensemble via Multisurface Proximal Support Vector Machine , 2015, IEEE Transactions on Cybernetics.

[15]  Nada Lavrac,et al.  Relating ensemble diversity and performance: A study in class noise detection , 2015, Neurocomputing.

[16]  Xin Yao,et al.  Diversity creation methods: a survey and categorisation , 2004, Inf. Fusion.

[17]  Xindong Wu,et al.  MReC4.5: C4.5 Ensemble Classification with MapReduce , 2009, 2009 Fourth ChinaGrid Annual Conference.

[18]  Eric Bauer,et al.  An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants , 1999, Machine Learning.

[19]  Eugene Semenkin,et al.  Neural networks ensemble approach for detecting attacks in computer networks , 2012, 2012 IEEE Congress on Evolutionary Computation.

[20]  Claudio Moraga,et al.  Training regression ensembles by sequential target correction and resampling , 2012, Inf. Sci..

[21]  P. Bühlmann.  Bagging, subagging and bragging for improving some prediction algorithms , 2003 .

[22]  Cynthia Rudin,et al.  The Rate of Convergence of Adaboost , 2011, COLT.

[23]  Yves Grandvalet,et al.  Bagging Equalizes Influence , 2004, Machine Learning.

[24]  Jon Crowcroft,et al.  A survey and comparison of peer-to-peer overlay network schemes , 2005, IEEE Communications Surveys & Tutorials.

[25]  Yaxin Bi.  The impact of diversity on the accuracy of evidential classifier ensembles , 2012, Int. J. Approx. Reason..

[26]  Joseph K. Bradley,et al.  FilterBoost: Regression and Classification on Large Datasets , 2007, NIPS.

[27]  Raúl Monge,et al.  Parallel Approach for Ensemble Learning with Locally Coupled Neural Networks , 2010, Neural Processing Letters.

[28]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[29]  Claudio Moraga,et al.  Two Bagging Algorithms with Coupled Learners to Encourage Diversity , 2007, IDA.

[30]  T. Poggio,et al.  Bagging Regularizes , 2002 .

[31]  Ponnuthurai N. Suganthan,et al.  Ensemble Classification and Regression-Recent Developments, Applications and Future Directions [Review Article] , 2016, IEEE Computational Intelligence Magazine.

[32]  D. Opitz,et al.  Popular Ensemble Methods: An Empirical Study , 1999, J. Artif. Intell. Res..

[33]  Cesare Furlanello,et al.  Parallelizing AdaBoost by weights dynamics , 2007, Comput. Stat. Data Anal..

[34]  Ponnuthurai N. Suganthan,et al.  Random Forests with ensemble of feature spaces , 2014, Pattern Recognit..

[35]  Lluís Màrquez i Villodre,et al.  Using LazyBoosting for Word Sense Disambiguation , 2001, *SEMEVAL.

[36]  Nikunj C. Oza,et al.  Online Ensemble Learning , 2000, AAAI/IAAI.

[37]  Ludmila I. Kuncheva,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2004 .

[38]  Stephen L. Olivier,et al.  Exploiting Geometric Partitioning in Task Mapping for Parallel Computers , 2014, 2014 IEEE 28th International Parallel and Distributed Processing Symposium.

[39]  Tianqi Chen,et al.  Higgs Boson Discovery with Boosted Trees , 2014, HEPML@NIPS.