A L-BFGS Based Learning Algorithm for Complex-Valued Feedforward Neural Networks

In this paper, a new learning algorithm is proposed for complex-valued feedforward neural networks (CVFNNs). The basic idea of this algorithm is that the descent directions of the cost function with respect to complex-valued parameters are calculated by limited-memory BFGS algorithm and the learning step is determined by Armijo line search method. Since the approximation of Hessian matrix is calculated by utilizing the information of the latest several iterations, the memory efficiency is improved. To keep away from the saturated ranges of activation functions, some gain parameters are adjusted together with weights and biases. Compared with some existing learning algorithms for CVFNNs, the convergence speed is faster and a deeper minima of the cost function can be reached by the developed algorithm. In addition, the effects of initial values of weights and biases on the efficiency and convergence speed of the learning algorithm are analyzed. The performance of the proposed algorithm is evaluated in comparison with some existing classifiers on a variety of benchmark classification problems. Experimental results show that better performance is achieved by our algorithm with relatively compact network structure.

[1]  Sundaram Suresh,et al.  A Fully Complex-valued Fast Learning Classifier (FC-FLC) for real-valued classification problems , 2015, Neurocomputing.

[2]  Xinghuo Yu,et al.  Distributed Optimal Consensus Over Resource Allocation Network and Its Application to Dynamical Economic Dispatch , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[3]  Chuandong Li,et al.  Robust Exponential Stability of Uncertain Delayed Neural Networks With Stochastic Perturbation and Impulse Effects , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Tülay Adali,et al.  Fully Complex Multi-Layer Perceptron Network for Nonlinear Signal Processing , 2002, J. VLSI Signal Process..

[5]  Modjtaba Rouhani,et al.  Two fast and accurate heuristic RBF learning rules for data classification , 2016, Neural Networks.

[6]  Akira Hirose,et al.  Complex-Valued Neural Networks: Advances and Applications , 2013 .

[7]  Xinghuo Yu,et al.  A Generalized Hopfield Network for Nonsmooth Constrained Convex Optimization: Lie Derivative Approach , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Ying Zhang,et al.  Boundedness and Convergence of Split-Complex Back-Propagation Algorithm with Momentum and Penalty , 2013, Neural Processing Letters.

[9]  Kazuyuki Murase,et al.  Classification of Skeletal Wireframe Representation of Hand Gesture Using Complex-Valued Neural Network , 2014, Neural Processing Letters.

[10]  Narasimhan Sundararajan,et al.  A Sequential Learning Algorithm for Complex-Valued Self-Regulating Resource Allocation Network-CSRAN , 2011, IEEE Transactions on Neural Networks.

[11]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[12]  Kazuyuki Murase,et al.  Single-layered complex-valued neural network for real-valued classification problems , 2009, Neurocomputing.

[13]  J. Nocedal.  Updating Quasi-Newton Matrices With Limited Storage , 1980 .

[14]  Edmondo Trentin,et al.  Networks with trainable amplitude of activation functions , 2001, Neural Networks.

[15]  Tohru Nitta.  The Computational Power of Complex-Valued Neuron , 2003, ICANN.

[16]  Cris Koutsougeras,et al.  Complex domain backpropagation , 1992 .

[17]  Akira Hirose,et al.  Generalization Characteristics of Complex-Valued Feedforward Neural Networks in Relation to Signal Coherence , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[18]  W. Rudin.  Real and complex analysis , 1968 .

[19]  N. Sundararajan,et al.  Complex-valued growing and pruning rbf neural networks for communication channel equalisation , 2006 .

[20]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[21]  Huisheng Zhang,et al.  A new adaptive momentum algorithm for split-complex recurrent neural networks , 2012, Neurocomputing.

[22]  Sundaram Suresh,et al.  Fast learning Circular Complex-valued Extreme Learning Machine (CC-ELM) for real-valued classification problems , 2012, Inf. Sci..

[23]  Rusli,et al.  Feedforward Neural Network Trained by BFGS Algorithm for Modeling Plasma Etching of Silicon Carbide , 2010, IEEE Transactions on Plasma Science.

[24]  Tohru Nitta,et al.  Orthogonality of Decision Boundaries in Complex-Valued Neural Networks , 2004, Neural Computation.

[25]  Jun Hu,et al.  A variance-constrained approach to recursive state estimation for time-varying complex networks with missing measurements , 2016, Autom..

[26]  J. J. Moré,et al.  Quasi-Newton Methods, Motivation and Theory , 1974 .

[27]  Calin-Adrian Popa.  Quasi-Newton learning methods for complex-valued neural networks , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[28]  Henry Leung,et al.  The complex backpropagation algorithm , 1991, IEEE Trans. Signal Process..

[29]  Tohru Nitta,et al.  Local minima in hierarchical structures of complex-valued neural networks , 2013, Neural Networks.

[30]  Sammy Siu,et al.  Sensitivity Analysis of the Split-Complex Valued Multilayer Perceptron Due to the Errors of the i.i.d. Inputs and Weights , 2007, IEEE Transactions on Neural Networks.

[31]  Stavros J. Perantonis,et al.  Two highly efficient second-order algorithms for training feedforward networks , 2002, IEEE Trans. Neural Networks.

[32]  Tingwen Huang,et al.  An improved maximum spread algorithm with application to complex-valued RBF neural networks , 2015 .

[33]  Jun Hu,et al.  Quantised recursive filtering for a class of nonlinear systems with multiplicative noises and missing measurements , 2013, Int. J. Control.

[34]  Ju-Jang Lee,et al.  Training Two-Layered Feedforward Networks With Variable Projection Method , 2008, IEEE Transactions on Neural Networks.

[35]  L. Armijo.  Minimization of functions having Lipschitz continuous first partial derivatives. , 1966 .