Bias of Estimators and Regularization Terms Noboru Murata

In this paper, a role of regularization terms (penalty terms) is discussed from the view point of minimizing the generalization error. First the bias of minimum training error estimation is clariied. The bias is caused by the nonlinearity of the learning system and depends on the number of training examples. Then an appropriate size of the regularization term is considered by taking account of the balance of the bias and the variance of the estimator, so that the generalization error is minimized. In this framework, the optimal size of the regularization term is calculated with the second and third order derivatives of the loss function. When the learning system has a large number of modiiable parameters, it is computationally expensive to calculate the higher order derivatives, thus we propose a simple method of approximating the optimal size via a generalized AIC.