Aggregation algorithms for neural network ensemble construction

How to generate and aggregate base learners to have optimal ensemble generalization capabilities is an important questions in building composite regression/classification machines. We present here an evaluation of several algorithms for artificial neural networks aggregation in the regression settings, including new proposals and comparing them with standard methods in the literature. We also discuss a potential problem with sequential algorithms: the non frequent but damaging selection through their heuristics of particularly bad ensemble members. We show that one can cope with this problem by allowing individual weighting of aggregate members. Our algorithms and their weighted modifications are favorably tested against other methods in the literature, producing a performance improvement on the standard statistical databases used as benchmarks.

[1]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[2]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[3]  Christopher J. Merz,et al.  UCI Repository of Machine Learning Databases , 1996 .

[4]  Harris Drucker,et al.  Improving Performance in Neural Networks Using a Boosting Algorithm , 1992, NIPS.

[5]  Nathan Intrator,et al.  Optimal ensemble averaging of neural networks , 1997 .

[6]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[7]  Harris Drucker,et al.  Improving Regressors using Boosting Techniques , 1997, ICML.

[8]  K. Ikeda.  Multiple-valued stationary state and its instability of the transmitted light by a ring cavity system , 1979 .

[9]  Amanda J. C. Sharkey,et al.  Boosting Using Neural Networks , 1999 .

[10]  Pablo M. Granitto,et al.  A Late-Stopping Method for Optimal Aggregation of Neural Networks , 2001, Int. J. Neural Syst..

[11]  JOHN G. CARNEY,et al.  Tuning Diversity in Bagged Ensembles , 2000, Int. J. Neural Syst..

[12]  Amanda J. C. Sharkey,et al.  On Combining Artificial Neural Nets , 1996, Connect. Sci..

[13]  Nathan Intrator,et al.  Boosting Regression Estimators , 1999, Neural Computation.

[14]  Amanda J. C. Sharkey,et al.  Combining Artificial Neural Nets: Ensemble and Modular Multi-Net Systems , 1999 .

[15]  Bruce E. Rosen,et al.  Ensemble Learning Using Decorrelated Neural Networks , 1996, Connect. Sci..

[16]  Pablo M. Granitto,et al.  A Learning Algorithm For Neural Network Ensembles , 2001, Inteligencia Artif..

[17]  John Shawe-Taylor,et al.  Towards a strategy for boosting regressors , 2000 .