On the regularization of forgetting recursive least square

In this paper, the regularization of employing the forgetting recursive least square (FRLS) training technique on feedforward neural networks is studied. We derive our result from the corresponding equations for the expected prediction error and the expected training error. By comparing these error equations with other equations obtained previously from the weight decay method, we have found that the FRLS technique has an effect which is identical to that of using the simple weight decay method. This new finding suggests that the FRLS technique is another on-line approach for the realization of the weight decay effect. Besides, we have shown that, under certain conditions, both the model complexity and the expected prediction error of the model being trained by the FRLS technique are better than the one trained by the standard RLS method.

[1]  S. Chen,et al.  Fast orthogonal least squares algorithm for efficient subset model selection , 1995, IEEE Trans. Signal Process..

[2]  L. K. Hansen,et al.  Pruning with generalization based weight saliencies: λOBD, λOBS , 1995, NIPS 1995.

[3]  Stefanos Kollias,et al.  An adaptive least squares algorithm for the efficient training of artificial neural networks , 1989 .

[4]  T. McKelveyDepartment.  On the Use of Regularization in System Identification , 1992 .

[5]  Christopher M. Bishop,et al.  Current address: Microsoft Research, , 2022 .

[6]  Sheng Chen,et al.  Practical identification of NARMAX models using radial basis functions , 1990 .

[7]  Tor Arne Johansen,et al.  On Tikhonov regularization, bias and variance in nonlinear system identification , 1997, Autom..

[8]  Chris Bishop,et al.  Improving the Generalization Properties of Radial Basis Function Neural Networks , 1991, Neural Computation.

[9]  Andrew Chi-Sing Leung,et al.  An Adaptive Bayesian Pruning for Neural Networks in a Non-Stationary Environment , 1999, Neural Computation.

[10]  Francesco Palmieri,et al.  Optimal filtering algorithms for fast learning in feedforward neural networks , 1992, Neural Networks.

[11]  L. Ljung,et al.  Overtraining, regularization and searching for a minimum, with application to neural networks , 1995 .

[12]  Dimitry M. Gorinevsky,et al.  On the persistency of excitation in radial basis function network identification of nonlinear systems , 1995, IEEE Trans. Neural Networks.

[13]  John Moody,et al.  Prediction Risk and Architecture Selection for Neural Networks , 1994 .

[14]  Kwok-Wo Wong,et al.  Periodic activation function for fast on-line EKF training and pruning , 1998 .

[15]  John E. Moody,et al.  The Effective Number of Parameters: An Analysis of Generalization and Regularization in Nonlinear Learning Systems , 1991, NIPS.

[16]  John Moody,et al.  Note on generalization, regularization and architecture selection in nonlinear learning systems , 1991, Neural Networks for Signal Processing Proceedings of the 1991 IEEE Workshop.

[17]  Rolf Johansson,et al.  System modeling and identification , 1993 .

[18]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[19]  Sheng Chen,et al.  Recursive hybrid algorithm for non-linear system identification using radial basis function networks , 1992 .

[20]  Lars Kai Hansen,et al.  Pruning with generalization based weight saliencies: gamma-OBD, gamma-OBS , 1995, NIPS.

[21]  Lennart Ljung,et al.  On the Use of Regularization in System Identification , 1993 .

[22]  Shun-ichi Amari,et al.  Network information criterion-determining the number of hidden units for an artificial neural network model , 1994, IEEE Trans. Neural Networks.

[23]  Sheng Chen,et al.  Parallel recursive prediction error algorithm for training layered neural networks , 1990 .

[24]  Lai-Wan Chan,et al.  On-line training and pruning for recursive least square algorithms , 1996 .

[25]  Sheng Chen,et al.  A clustering technique for digital communications channel equalization using radial basis function networks , 1993, IEEE Trans. Neural Networks.

[26]  Lars Kai Hansen,et al.  Generalization performance of regularized neural network models , 1994, Proceedings of IEEE Workshop on Neural Networks for Signal Processing.

[27]  Lizhong Wu,et al.  A Smoothing Regularizer for Feedforward and Recurrent Neural Networks , 1996, Neural Computation.