Bias-variance, regularization, instability and stabilization

This chapter is concerned with some of the fundamentals in predicting numerical outcomes - known in statistics as regression. Here is a road map: Using the test set definition of mean-squared prediction error, we will see that the prediction error can be decomposed into two major components - bias and variance. The idea of regularization is to construct a sequence of predictors that begin with high variance-low bias and go to low variance-high bias. Then the problem becomes selecting the member of this sequence having the lowest bias-variance sum. How well we are able to do this turns out to depend on the stability of the method for constructing the predictors. Many methods, including decision trees and neural nets are unstable. But unstable procedures can be stabilized leading to significant improvements in accuracy.