Improving neural networks generalization with new constructive and pruning methods

This paper presents a new constructive method and pruning approaches to control the design of Multi-Layer Perceptron (MLP) without loss in performance. The proposed methods use a multi-objective approach to guarantee generalization. The constructive approach searches for an optimal solution according to the pareto set shape with increasing number of hidden nodes. The pruning methods are able to simplify the network topology and to identify linear connections between the inputs and outputs of the neural model. Topology information and validation sets are used.

[1]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[2]  David E. Rumelhart,et al.  Predicting the Future: a Connectionist Approach , 1990, Int. J. Neural Syst..

[3]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[4]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[5]  Ricardo H. C. Takahashi,et al.  Improving generalization of MLPs with multi-objective optimization , 2000, Neurocomputing.

[6]  Antônio de Pádua Braga,et al.  Training neural networks with a multi-objective sliding mode control algorithm , 2003, Neurocomputing.

[7]  J. Nazuno.  Haykin, Simon. Neural networks: A comprehensive foundation, Prentice Hall, Inc. Segunda Edición, 1999 , 2000 .

[8]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[9]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[10]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[11]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[12]  Peter L. Bartlett,et al.  For Valid Generalization the Size of the Weights is More Important than the Size of the Network , 1996, NIPS.

[13]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[14]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[15]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.