Learning Algorithms in Quaternion Neural Networks Using GHR Calculus

One difficulty for quaternion neural networks (QNNs) is that quaternion nonlinear activation functions are usually non-analytic and thus quaternion derivatives cannot be used. In this paper, we derive the quaternion gradient descent, approximated quaternion Gauss-Newton and quaternion Levenberg-Marquardt algorithms for feedforward QNNs based on the GHR calculus, which is suitable for analytic and non-analytic quaternion functions. Meanwhile, we solve a widely linear quaternion least squares problem in the derivation of quaternion Gauss-Newton algorithm, which is more general than the usual least squares problem. A rigorous analysis of the convergence of the proposed algorithms is provided. Simulations on the prediction of benchmark signals support the approach.

[1]  W. Wirtinger.  Zur formalen Theorie der Funktionen von mehr komplexen Veränderlichen , 1927 .

[2]  C. Deavours.  The Quaternion Calculus , 1973 .

[3]  B. A. D. H. Brandwood.  A complex gradient operator and its applica-tion in adaptive array theory , 1983 .

[4]  Toshimichi Saito,et al.  Torus doubling and hyperchaos in a five dimensional hysteresis circuit , 1994, Proceedings of IEEE International Symposium on Circuits and Systems - ISCAS '94.

[5]  Giovanni Muscato,et al.  Multilayer Perceptrons to Approximate Quaternion Valued Functions , 1997, Neural Networks.

[6]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[7]  Stefano De Leo,et al.  Quaternionic analyticity , 2003, Appl. Math. Lett..

[8]  Nobuyuki Matsui,et al.  Quaternion neural network with geometrical operators , 2004, J. Intell. Fuzzy Syst..

[9]  Fuzhen Zhang.  The Schur complement and its applications , 2005 .

[10]  Stephen J. Sangwine,et al.  Quaternion involutions and anti-involutions , 2005, Comput. Math. Appl..

[11]  Nicolas Le Bihan,et al.  Polarized Signal Classification by Complex and quaternionic Multi-Layer Perceptrons , 2008, Int. J. Neural Syst..

[12]  Danilo P. Mandic,et al.  The Quaternion LMS Algorithm for Adaptive Filtering of Hypercomplex Processes , 2009, IEEE Transactions on Signal Processing.

[13]  Ignacio Santamaría,et al.  Properness and Widely Linear Processing of Quaternion Random Vectors , 2010, IEEE Transactions on Information Theory.

[14]  Danilo P. Mandic,et al.  Split quaternion nonlinear adaptive filtering , 2010, Neural Networks.

[15]  Danilo P. Mandic,et al.  Augmented second-order statistics of quaternion random signals , 2011, Signal Process..

[16]  Danilo P. Mandic,et al.  A Quaternion Gradient Operator and Its Applications , 2011, IEEE Signal Processing Letters.

[17]  Danilo P. Mandic,et al.  A Class of Quaternion Kalman Filters , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Danilo P. Mandic,et al.  Performance Bounds of Quaternion Estimators , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[19]  Danilo P. Mandic,et al.  Quaternion-Valued Echo State Networks , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[20]  Danilo P. Mandic,et al.  The Theory of Quaternion Matrix Derivatives , 2014, IEEE Transactions on Signal Processing.

[21]  Dongpo Xu,et al.  Enabling quaternion derivatives: the generalized HR calculus , 2014, Royal Society Open Science.

[22]  Danilo P. Mandic,et al.  Optimization in Quaternion Dynamic Systems: Gradient, Hessian, and Learning Algorithms , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[23]  Danilo P. Mandic,et al.  A new proof of the generalized Hamiltonian–Real calculus , 2016, Royal Society Open Science.