Hyperbolic Gradient Operator and Hyperbolic Back-Propagation Learning Algorithms

In this paper, we first extend the Wirtinger derivative which is defined for complex functions to hyperbolic functions, and derive the hyperbolic gradient operator yielding the steepest descent direction by using it. Next, we derive the hyperbolic backpropagation learning algorithms for some multilayered hyperbolic neural networks (NNs) using the hyperbolic gradient operator. It is shown that the use of the Wirtinger derivative reduces the effort necessary for the derivation of the learning algorithms by half, simplifies the representation of the learning algorithms, and makes their computer programs easier to code. In addition, we discuss the differences between the derived Hyperbolic-BP rules and the complex-valued backpropagation learning rule (Complex-BP). Finally, we make some experiments with the derived learning algorithms. As a result, we find that the convergence rates of the Hyperbolic-BP learning algorithms are high even if the fully activation functions are used, and discover that the Hyperbolic-BP learning algorithm for the hyperbolic NN with the split-type hyperbolic activation function has an ability to learn hyperbolic rotation as its inherent property.

[1]  Akira Watanabe,et al.  A Method to Interpret 3D Motions Using Neural Networks (Special Section on Information Theory and Its Applications) , 1994 .

[2]  Masaki Kobayashi,et al.  Hyperbolic Hopfield Neural Networks , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[3]  Akira Hirose,et al.  Complex-Valued Neural Networks: Advances and Applications , 2013 .

[4]  Danilo P. Mandic,et al.  A Quaternion Gradient Operator and Its Applications , 2011, IEEE Signal Processing Letters.

[5]  W. Wirtinger.  Zur formalen Theorie der Funktionen von mehr komplexen Veränderlichen , 1927 .

[6]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[7]  Helge J. Ritter,et al.  Hyperbolic Self-Organizing Maps for Semantic Navigation , 2001, NIPS.

[8]  Garret Sobczyk,et al.  The Hyperbolic Number Plane , 1995 .

[9]  Clark C. Guest,et al.  Modification of backpropagation networks for complex-valued signal processing in frequency domain , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[10]  B. Palka.  An Introduction to Complex Function Theory , 1995 .

[11]  A. Motter,et al.  Hyperbolic Calculus , 1998 .

[12]  Danilo P. Mandic,et al.  Complex Valued Nonlinear Adaptive Filters , 2009 .

[13]  Kazuyuki Murase,et al.  Wirtinger Calculus Based Gradient Descent and Levenberg-Marquardt Learning Algorithms in Complex-Valued Neural Networks , 2011, ICONIP.

[14]  B. A. D. H. Brandwood.  A complex gradient operator and its applica-tion in adaptive array theory , 1983 .

[15]  Tohru Nitta,et al.  Orthogonality of Decision Boundaries in Complex-Valued Neural Networks , 2004, Neural Computation.

[16]  Jonas Schreiber.  Complex Valued Neural Networks Utilizing High Dimensional Parameters , 2016 .

[17]  Gerald Sommer,et al.  A hyperbolic multilayer perceptron , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[18]  Nobuyuki Matsui,et al.  Quaternionic Multilayer Perceptron with Local Analyticity , 2012, Inf..

[19]  Tohru Nitta,et al.  On the decision boundaries of hyperbolic neurons , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[20]  Eckhard Hitzer.  Non-constant bounded holomorphic functions of hyperbolic numbers - Candidates for hyperbolic activation functions , 2013, ArXiv.

[21]  Tohru Nitta,et al.  An Extension of the Back-Propagation Algorithm to Complex Numbers , 1997, Neural Networks.

[22]  Yasuaki Kuroe,et al.  Models of Hopfield-Type Clifford Neural Networks and Their Energy Functions - Hyperbolic and Dual Valued Networks - , 2011, ICONIP.

[23]  Hilary A. Priestley,et al.  Introduction to Complex Analysis , 1985 .

[24]  H. Ritter.  Self-Organizing Maps on non-euclidean Spaces , 1999 .

[25]  Ken Kreutz-Delgado,et al.  The Complex Gradient Operator and the CR-Calculus ECE275A - Lecture Supplement - Fall 2005 , 2009, 0906.4835.

[26]  Helge J. Ritter,et al.  Text Categorization and Semantic Browsing with Self-Organizing Maps on Non-euclidean Spaces , 2001, PKDD.