Regularization versus early stopping: a case study with a real system

Regularization and Early Stopping are two of the most common techniques to deal with the overtraining problem in the Artificial Neural Networks field. The overtraining problem appears mostly in systems affected by noise in which after a certain amount of training, the neural network used for modelling starts to learn information specific from the training signal or the noise. It has already been shown that these techniques can be used to avoid this problem and they are formerly equivalent, but this issue deserve further investigation since real systems sometimes behave in a different way than simulated systems. A fair comparison for the two techniques is not very easy to make since in the networks there are several parameters that cannot be determined in an analytical way. To overcome this difficulty in the present work a procedure for automating the construction of the models has been used. This procedure allows creating models that are optimised in the number of inputs, the number of hidden neurons and the generalization capability using either Early Stopping or Regularization. This enables the possibility of performing a fair comparison. The procedure includes an hybrid direct/specialized training solution for evaluating the inverse model. To test the results the system used is a reduced scale prototype kiln affected by measurement noise, for which the Direct Inverse Control and Internal Model Control strategies were implemented.

[1]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[2]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[3]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[4]  Daniel George Haesloop.  System identification and control using neural networks with combined linear and non-linear mapping functionality , 1991 .

[5]  Peter J. Gawthrop,et al.  Neural networks for control systems - A survey , 1992, Autom..

[6]  L. Ljung,et al.  Overtraining, Regularization, and Searching for Minimum in Neural Networks , 1992 .

[7]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[8]  O. Sørensen.  Neural Networks in Control Applications , 1994 .

[9]  Lars Kai Hansen,et al.  Recurrent Networks: Second Order Properties and Pruning , 1994, NIPS.

[10]  J. Sjöberg.  Non-Linear System Identification with Neural Networks , 1995 .

[11]  George W. Irwin,et al.  Nonlinear control structures based on embedded neural system models , 1997, IEEE Trans. Neural Networks.

[12]  Alexandre Mota,et al.  Comparison between different Control Strategies using Neural Networks , 2001 .

[13]  Alexandre Mota,et al.  A Comparison between a PID and Internal Model Control using Neural Networks , 2001 .

[14]  Fernando Morgado Dias,et al.  Automating the construction of neural models for control purposes using genetic algorithms , 2002, IEEE 2002 28th Annual Conference of the Industrial Electronics Society. IECON 02.