OBSERVATIONS ON BAGGING

Bagging is a device intended for reducing the prediction error of learning algorithms. In its simplest form, bagging draws bootstrap samples from the training sample, applies the learning algorithm to each bootstrap sample, and then averages the resulting prediction rules. More generally, the resample size M may be dieren t from the original sample size N, and resampling can be done with or without replacement. We investigate bagging in a simplied situation: the prediction rule produced by a learning algorithm is replaced by a simple real-valued U-statistic of i.i.d. data. U-statistics of high order can describe complex dependencies, and yet they admit a rigorous asymptotic analysis. We show that bagging U-statistics often but not always decreases variance, whereas it always increases bias. The most striking nding, however, is an equivalence between bagging based on resampling with and without replacement: the respective resample sizes Mwith = withN and Mw=o = w=oN produce very similar bagged statistics if with = w=o=(1 w=o). While our derivation is limited to U-statistics, the equivalence seems to be universal. We illustrate this point in simulations where bagging is applied to cart trees.