Common nature of learning between BP-type and Hopfield-type neural networks

Being two famous neural networks, the error back-propagation (BP) algorithm based neural networks (i.e., BP-type neural networks, BPNNs) and Hopfield-type neural networks (HNNs) have been proposed, developed, and investigated extensively for scientific research and engineering applications. They are different from each other in a great deal, in terms of network architecture, physical meaning and training pattern. In this paper of literature-review type, we present in a relatively complete and creative manner the common natures of learning between BP-type and Hopfield-type neural networks for solving various (mathematical) problems. Specifically, comparing the BPNN with the HNN for the same problem-solving task, e.g., matrix inversion as well as function approximation, we show that the BPNN weight-updating formula and the HNN state-transition equation turn out to be essentially the same. Such interesting phenomena promise that, given a neural-network model for a specific problem solving, its potential dual neural-network model can thus be developed.

[1]  Wanbiao Ma,et al.  Sufficient conditions for global attractivity of a class of neutral Hopfield-type neural networks , 2015, Neurocomputing.

[2]  Dianhui Wang,et al.  Global Convergence of Online BP Training With Dynamic Learning Rate , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[3]  Yongguang Yu,et al.  Mittag-Leffler stability of fractional-order Hopfield neural networks , 2015 .

[4]  Udo Seiffert,et al.  ANNIE - Artificial Neural Network-based Image Encoder , 2014, Neurocomputing.

[5]  Chin-Hui Lee,et al.  An artificial neural network approach to automatic speech processing , 2014, Neurocomputing.

[6]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[7]  Chenglie Du,et al.  An integrated micromechanical model and BP neural network for predicting elastic modulus of 3-D multi-phase and multi-layer braided composite , 2015 .

[8]  Yanxi Zhang,et al.  Weld appearance prediction with BP neural network improved by genetic algorithm during disk laser welding , 2015 .

[9]  Zhihong Man,et al.  A New Adaptive Backpropagation Algorithm Based on Lyapunov Stability Theory for Neural Networks , 2006, IEEE Transactions on Neural Networks.

[10]  Gul Muhammad Khan,et al.  Fast learning neural networks using Cartesian genetic programming , 2013, Neurocomputing.

[11]  Han-Xiong Huang,et al.  A proposed iteration optimization approach integrating backpropagation neural network with genetic algorithm , 2015, Expert Syst. Appl..

[12]  Jinde Cao,et al.  A Novel Recurrent Neural Network With One Neuron and Finite-Time Convergence for $k$-Winners-Take-All Operation , 2010, IEEE Transactions on Neural Networks.

[13]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[14]  Jinde Cao,et al.  Existence and Uniform Stability Analysis of Fractional-Order Complex-Valued Neural Networks With Time Delays , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Haijun Jiang,et al.  Projective synchronization for fractional neural networks , 2014, Neural Networks.

[16]  Plamen P. Angelov,et al.  Uniformly Stable Backpropagation Algorithm to Train a Feedforward Neural Network , 2011, IEEE Transactions on Neural Networks.

[17]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[18]  Junzhi Yu,et al.  Global stability analysis of fractional-order Hopfield neural networks with time delay , 2015, Neurocomputing.

[19]  Binghuang Cai,et al.  From Zhang Neural Network to Newton Iteration for Matrix Inversion , 2009, IEEE Transactions on Circuits and Systems I: Regular Papers.

[20]  J. Hopfield,et al.  Computing with neural circuits: a model. , 1986, Science.

[21]  Gouhei Tanaka.  Complex-Valued Neural Networks: Advances and Applications [Book Review] , 2013, IEEE Computational Intelligence Magazine.

[22]  Zhijun Zhang,et al.  Weights and structure determination of multiple-input feed-forward neural network activated by Chebyshev polynomials of Class 2 via cross-validation , 2014, Neural Computing and Applications.

[23]  Fuzhi Zhang,et al.  Ensemble detection model for profile injection attacks in collaborative recommender systems based on BP neural network , 2015, IET Inf. Secur..

[24]  Yunong Zhang,et al.  Zhang Neural Networks and Neural-Dynamic Method , 2011 .

[25]  Dongsheng Guo,et al.  Zhang neural network, Getz-Marsden dynamic system, and discrete-time algorithms for time-varying matrix inversion with application to robots' kinematic control , 2012, Neurocomputing.

[26]  Feng Yu,et al.  A short-term load forecasting model of natural gas based on optimized genetic algorithm and improved BP neural network , 2014 .

[27]  A. Sadeghian,et al.  On the use of recurrent neural networks to design symmetric ciphers , 2008, IEEE Computational Intelligence Magazine.

[28]  Raja Muhammad Asif Zahoor,et al.  Numerical treatment for nonlinear MHD Jeffery-Hamel problem using neural networks optimized with interior point algorithm , 2014, Neurocomputing.

[29]  Jimi Tjong,et al.  Automotive Internal-Combustion-Engine Fault Detection and Classification Using Artificial Neural Network Techniques , 2015, IEEE Transactions on Vehicular Technology.

[30]  Yunong Zhang,et al.  A new variant of the Zhang neural network for solving online time-varying linear inequalities , 2012, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[31]  Dongsheng Guo,et al.  Novel Recurrent Neural Network for Time-Varying Problems Solving [Research Frontier] , 2012, IEEE Computational Intelligence Magazine.

[32]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[33]  Wuneng Zhou,et al.  Finite-time state estimation for delayed Hopfield neural networks with Markovian jump , 2015, Neurocomputing.

[34]  Dongsheng Guo,et al.  Common Nature of Learning Between Back-Propagation and Hopfield-Type Neural Networks for Generalized Matrix Inversion With Simplified Models , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[35]  Yunong Zhang,et al.  Euler Neural Network with Its Weight-Direct-Determination and Structure-Automatic-Determination Algorithms , 2009, 2009 Ninth International Conference on Hybrid Intelligent Systems.

[36]  George A. F. Seber,et al.  A matrix handbook for statisticians , 2007 .

[37]  Jinde Cao,et al.  Dynamics in fractional-order neural networks , 2014, Neurocomputing.

[38]  George Lindfield,et al.  Numerical Methods Using MATLAB , 1998 .

[39]  Fei Wang,et al.  Asymptotic stability of delayed fractional-order neural networks with impulsive effects , 2015, Neurocomputing.

[40]  Ke Chen,et al.  Performance Analysis of Gradient Neural Network Exploited for Online Time-Varying Matrix Inversion , 2009, IEEE Transactions on Automatic Control.

[41]  Akira Hirose,et al.  Complex-Valued Neural Networks: Advances and Applications , 2013 .

[42]  Adnan Khashman,et al.  A Modified Backpropagation Learning Algorithm With Added Emotional Coefficients , 2008, IEEE Transactions on Neural Networks.

[43]  Dongsheng Guo,et al.  Cross-validation based weights and structure determination of Chebyshev-polynomial neural networks for pattern classification , 2014, Pattern Recognit..

[44]  Wen-Ching Chen,et al.  A unified evolutionary training scheme for single and ensemble of feedforward neural network , 2014, Neurocomputing.

[45]  Jean-Luc Buessler,et al.  Image receptive fields for artificial neural networks , 2014, Neurocomputing.

[46]  José de Jesús Rubio,et al.  Backpropagation to train an evolving radial basis function neural network , 2010, Evol. Syst..

[47]  Akira Hirose,et al.  Complex-Valued Neural Networks , 2006, Studies in Computational Intelligence.