Modified constrained learning algorithms incorporating additional functional constraints into neural networks

In this paper, two modified constrained learning algorithms are proposed to obtain better generalization performance and faster convergence rate. The additional cost terms of the first algorithm are selected based on the first-order derivatives of the activation functions of the hidden neurons and the second-order derivatives of the activation functions of the output neurons, while the additional cost terms of the second one are selected based on the first-order derivatives of the activation functions of the output neurons and the second-order derivatives of the activation functions of the hidden neurons. In the course of training, the additional cost terms of the proposed algorithms can penalize the input-to-output mapping sensitivity and the high frequency components simultaneously so that the better generalization performance can be obtained. Finally, theoretical justifications and simulation results are given to verify the efficiency and effectiveness of our proposed learning algorithms.

[1]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[2]  A Learning Algorithm for Fault Tolerant Feedforward Neural Networks , 1996 .

[3]  Soo-Young Lee,et al.  Adaptive learning algorithms to incorporate additional functional constraints into neural networks , 1998, Neurocomputing.

[4]  Lars Kai Hansen,et al.  Regularization with a Pruning Prior , 1997, Neural Networks.

[5]  Ehud D. Karnin,et al.  A simple procedure for pruning back-propagation trained neural networks , 1990, IEEE Trans. Neural Networks.

[6]  T Poggio,et al.  Regularization Algorithms for Learning That Are Equivalent to Multilayer Networks , 1990, Science.

[7]  Jiwen Dong,et al.  Time-series forecasting using flexible neural tree model , 2005, Inf. Sci..

[8]  D. Rumelhart,et al.  Generalization by weight-elimination applied to currency exchange rate prediction , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[9]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[10]  Marie Cottrell,et al.  Neural modeling for time series: A statistical stepwise method for weight elimination , 1995, IEEE Trans. Neural Networks.

[11]  De-Shuang Huang,et al.  A Neural Root Finder of Polynomials Based on Root Moments , 2004, Neural Computation.

[12]  De-Shuang Huang,et al.  A constructive approach for finding arbitrary roots of polynomials by neural networks , 2004, IEEE Transactions on Neural Networks.

[13]  De-Shuang Huang,et al.  Finding roots of arbitrary high order polynomials based on neural network recursive partitioning method , 2007, Science in China Series F: Information Sciences.

[14]  Soo-Young Lee,et al.  Merging Back-propagation and Hebbian Learning Rules for Robust Classifications , 1996, Neural Networks.

[15]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[16]  Young Joon Choi,et al.  Hybrid accident simulation methodology using artificial neural networks for nuclear power plants , 2004, Inf. Sci..

[17]  Christopher M. Bishop,et al.  Curvature-driven smoothing: a learning algorithm for feedforward networks , 1993, IEEE Trans. Neural Networks.

[18]  Geoffrey E. Hinton,et al.  Phoneme recognition using time-delay neural networks , 1989, IEEE Trans. Acoust. Speech Signal Process..

[19]  Michele Marchesi,et al.  A hybrid genetic-neural architecture for stock indexes forecasting , 2005, Inf. Sci..

[20]  De-Shuang Huang,et al.  Dilation method for finding close roots of polynomials based on constrained learning neural networks , 2003 .

[21]  Peter M. Williams,et al.  Bayesian Regularization and Pruning Using a Laplace Prior , 1995, Neural Computation.

[22]  Dimitris A. Karras,et al.  An efficient constrained training algorithm for feedforward networks , 1995, IEEE Trans. Neural Networks.

[23]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.

[24]  Geoffrey E. Hinton,et al.  Simplifying Neural Networks by Soft Weight-Sharing , 1992, Neural Computation.

[25]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[26]  Rey-Chue Hwang,et al.  An effective learning of neural network by using RFBP learning algorithm , 2004, Inf. Sci..

[27]  Ignacio Rojas,et al.  Improving the tolerance of multilayer perceptrons by minimizing the statistical sensitivity to weight deviations , 2000, Neurocomputing.