Combining Stacking With Bagging To Improve A Learning Algorithm

In bagging Bre94a] one uses bootstrap replicates of the training set Efr79, ET93] to improve a learning algorithm's performance, often by tens of percent. This paper presents several ways that stacking Wol92b, Bre92] can be used in concert with the bootstrap procedure to achieve a further improvement on the performance of bagging for some regression problems. In particular, in some of the work presented here, one rst converts a single underlying learning algorithm into several learning algorithms. This is done by bootstrap resampling the training set, exactly as in bagging. The resultant algorithms are then combined via stacking. This procedure can be viewed as a variant of bagging, where stacking rather than uniform averaging is used to achieve the combining. The stacking improves performance over simple bagging by up to a factor of 2 on the tested problems, and never resulted in worse performance than simple bagging. In other work presented here, there is no step of converting the underlying learning algorithm into multiple algorithms, so it is the improve-a-single-algorithm variant of stacking that is relevant. The precise version of this scheme tested can be viewed as using the bootstrap and stacking to estimate the input-dependence of the statistical bias and then correct for it. The results are preliminary, but again indicate that combining stacking with the bootstrap can be helpful.