A yet faster version of complex-valued multilayer perceptron learning using singular regions and search pruning

In the search space of a complex-valued multilayer perceptron having J hidden units, C-MLP(J), there are singular regions, where the gradient is zero. Although singular regions cause serious stagnation of learning, there exist narrow descending paths from the regions. Based on this observation, a completely new learning method called C-SSF (complex singularity stairs following) 1.0 was proposed, which utilizes singular regions to generate starting points of C-MLP(J) search. Although C-SSF1.0 finds excellent solutions of successive C-MLPs, it takes long CPU time because the number of searches increases as J gets larger. To deal with this problem, C-SSF1.1 was proposed, a few times faster by the introduction of search pruning, but it still remained unsatisfactory. In this paper we propose a yet faster C-SSF1.3, going further with search pruning, and then evaluate the method in terms of solution quality and processing time.

[1]  R. Nakano,et al.  A yet faster version of complex-valued multilayer perceptron learning using singular regions and search pruning , 2015, IJCCI 2015.

[2]  Akira Hirose,et al.  Complex-Valued Neural Networks , 2006, Studies in Computational Intelligence.

[3]  Ryohei Nakano,et al.  Fast and Stable Learning Utilizing Singular Regions of Multilayer Perceptron , 2013, Neural Processing Letters.

[4]  Ken Kreutz-Delgado,et al.  The Complex Gradient Operator and the CR-Calculus ECE275A - Lecture Supplement - Fall 2005 , 2009, 0906.4835.

[5]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[6]  Kazuyuki Murase,et al.  Wirtinger Calculus Based Gradient Descent and Levenberg-Marquardt Learning Algorithms in Complex-Valued Neural Networks , 2011, ICONIP.

[7]  Héctor J. Sussmann,et al.  Uniqueness of the weights for minimal feedforward nets with a given input-output map , 1992, Neural Networks.

[8]  R. Remmert,et al.  Theory of Complex Functions , 1990 .

[9]  Clark C. Guest,et al.  Modification of backpropagation networks for complex-valued signal processing in frequency domain , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[10]  S. Suzumura,et al.  Complex-Valued BFGS Method for Complex-Valued Neural Networks , 2013 .

[11]  Tohru Nitta,et al.  Reducibility of the Complex-valued Neural Network , 2004 .

[12]  Ryohei Nakano,et al.  Complex-Valued Multilayer Perceptron Search Utilizing Singular Regions of Complex-Valued Parameter Space , 2014, ICANN.

[13]  Ryohei Nakano,et al.  Learning Method Utilizing Singular Region of Multilayer Perceptron , 2011, IJCCI.

[14]  Henry Leung,et al.  The complex backpropagation algorithm , 1991, IEEE Trans. Signal Process..

[15]  Tohru Nitta,et al.  Local minima in hierarchical structures of complex-valued neural networks , 2013, Neural Networks.

[16]  Kenji Fukumizu,et al.  Local minima and plateaus in hierarchical structures of multilayer perceptrons , 2000, Neural Networks.

[17]  Phillipp Kaestner,et al.  Linear And Nonlinear Programming , 2016 .

[18]  Satoh Seiya,et al.  Eigen Vector Descent and Line Search for Multilayer Perceptron , 2011 .