A survey of quaternion neural networks

Quaternion neural networks have recently received an increasing interest due to noticeable improvements over real-valued neural networks on real world tasks such as image, speech and signal processing. The extension of quaternion numbers to neural architectures reached state-of-the-art performances with a reduction of the number of neural parameters. This survey provides a review of past and recent research on quaternion neural networks and their applications in different domains. The paper details methods, algorithms and applications for each quaternion-valued neural networks proposed.

[1]  W. Hamilton.  II. On quaternions; or on a new system of imaginaries in algebra , 1844 .

[2]  Ken Shoemake,et al.  Animating rotation with quaternion curves , 1985, SIGGRAPH.

[3]  N. Higham.  Analysis of the Cholesky Decomposition of a Semi-definite Matrix , 1990 .

[4]  J.C.K. Chou,et al.  Quaternion kinematic and dynamic differential equations , 1992, IEEE Trans. Robotics Autom..

[5]  Luigi Fortuna,et al.  On the capability of neural networks with complex neurons in complex valued functions approximation , 1993, 1993 IEEE International Symposium on Circuits and Systems.

[6]  Alan F. Murray,et al.  International Joint Conference on Neural Networks , 1993 .

[7]  Jonathan G. Fiscus,et al.  DARPA TIMIT:: acoustic-phonetic continuous speech corpus CD-ROM, NIST speech disc 1-1.1 , 1993 .

[8]  Luigi Fortuna,et al.  Neural networks for quaternion-valued function approximation , 1994, Proceedings of IEEE International Symposium on Circuits and Systems - ISCAS '94.

[9]  T. Nitta,et al.  A quaternary version of the back-propagation algorithm , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[10]  Giovanni Muscato,et al.  An hypercomplex neural network platform for robot positioning , 1996, 1996 IEEE International Symposium on Circuits and Systems. Circuits and Systems Connecting the World. ISCAS 96.

[11]  Jzau-Sheng Lin,et al.  A fuzzy Hopfield neural network for medical image segmentation , 1996 .

[12]  S. Sangwine.  Fourier transforms of colour images using quaternion or hypercomplex, numbers , 1996 .

[13]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[14]  S. Leo,et al.  Local Hypercomplex Analyticity , 1997, funct-an/9703002.

[15]  Giovanni Muscato,et al.  Multilayer Perceptrons to Approximate Quaternion Valued Functions , 1997, Neural Networks.

[16]  Kuldip K. Paliwal,et al.  Bidirectional recurrent neural networks , 1997, IEEE Trans. Signal Process..

[17]  Marti A. Hearst.  Trends & Controversies: Support Vector Machines , 1998, IEEE Intell. Syst..

[18]  Fadi Dornaika,et al.  Simultaneous robot-world and hand-eye calibration , 1998, IEEE Trans. Robotics Autom..

[19]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[20]  Gerald Sommer,et al.  Quaternionic spinor MLP , 2000, ESANN.

[21]  Giovanni Muscato,et al.  A comparison between HMLP and HRBF for attitude control , 2001, IEEE Trans. Neural Networks.

[22]  Heiga Zen,et al.  Trajectory modeling based on HMMs with the explicit relationship between static and dynamic features , 2003, INTERSPEECH.

[23]  Nobuyuki Matsui,et al.  Quaternion Neural Network and Its Application , 2003, KES.

[24]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[25]  Nobuyuki Matsui,et al.  Quaternion neural network with geometrical operators , 2004, J. Intell. Fuzzy Syst..

[26]  Hiromi Kusamichi,et al.  A New Scheme for Color Night Vision by Quaternion Neural Network , 2004 .

[27]  T. Nitta,et al.  A Solution to the 4-bit Parity Problem with a Single Quaternary Neuron , 2004 .

[28]  J. J. Hopfield,et al.  “Neural” computation of decisions in optimization problems , 1985, Biological Cybernetics.

[29]  C. Willmott,et al.  Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance , 2005 .

[30]  Daniel Pletinckx,et al.  Quaternion calculus as a basic tool in computer graphics , 2005, The Visual Computer.

[31]  Yasuaki Kuroe,et al.  Models of hopfield-type quaternion neural networks and their energy functions , 2005, Int. J. Neural Syst..

[32]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[33]  Shie Mannor,et al.  A Tutorial on the Cross-Entropy Method , 2005, Ann. Oper. Res..

[34]  Nicolas Le Bihan,et al.  Optimal separation of polarized signals by quaternionic neural networks , 2006, 2006 14th European Signal Processing Conference.

[35]  Xiaoping Yun,et al.  Design, Implementation, and Experimental Results of a Quaternion-Based Kalman Filter for Human Body Motion Tracking , 2006, IEEE Trans. Robotics.

[36]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[37]  James Diebel,et al.  Representing Attitude : Euler Angles , Unit Quaternions , and Rotation Vectors , 2006 .

[38]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[39]  Naotake Kamiura,et al.  Fundamental Properties of Quaternionic Hopfield Neural Network , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[40]  Yann LeCun,et al.  Large-scale Learning with SVM and Convolutional for Generic Object Categorization , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[41]  Charles F. F. Karney.  Quaternions in molecular modeling. , 2005, Journal of molecular graphics & modelling.

[42]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[43]  Naotake Kamiura,et al.  Associative Memory in quaternionic Hopfield Neural Network , 2008, Int. J. Neural Syst..

[44]  Danilo P. Mandic,et al.  A split quaternion nonlinear adaptive filter , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[45]  Tohru Nitta.  Complex-valued Neural Networks: Utilizing High-dimensional Parameters , 2009 .

[46]  D. Mandic,et al.  Complex Valued Nonlinear Adaptive Filters: Noncircularity, Widely Linear and Neural Models , 2009 .

[47]  Nobuyuki Matsui,et al.  Quaternionic Neural Networks: Fundamental Properties and Applications , 2009 .

[48]  Lakhmi C. Jain,et al.  Knowledge-Based and Intelligent Information and Engineering Systems , 2011, Lecture Notes in Computer Science.

[49]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[50]  Danilo P. Mandic,et al.  Complex Valued Nonlinear Adaptive Filters , 2009 .

[51]  Lukás Burget,et al.  Recurrent neural network based language model , 2010, INTERSPEECH.

[52]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[53]  Philippe Carré,et al.  Quaternionic wavelets for texture classification , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[54]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[55]  D. Mandic,et al.  Quaternion Valued Neural Networks and Nonlinear Adaptive Filters ∗ , 2010 .

[56]  Lukás Burget,et al.  Extensions of recurrent neural network language model , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[57]  Igor N. Aizenberg,et al.  Recognition of Blurred Images Using Multilayer Neural Network Based on Multi-valued Neurons , 2011, 2011 41st IEEE International Symposium on Multiple-Valued Logic.

[58]  Danilo P. Mandic,et al.  A Quaternion Gradient Operator and Its Applications , 2011, IEEE Signal Processing Letters.

[59]  Miodrag Lovric,et al.  International Encyclopedia of Statistical Science , 2011 .

[60]  Danilo P. Mandic,et al.  Quaternion-Valued Nonlinear Adaptive Filtering , 2011, IEEE Transactions on Neural Networks.

[61]  Ian T. Jolliffe,et al.  Principal Component Analysis , 2002, International Encyclopedia of Statistical Science.

[62]  Tara N. Sainath,et al.  FUNDAMENTAL TECHNOLOGIES IN MODERN SPEECH RECOGNITION Digital Object Identifier 10.1109/MSP.2012.2205597 , 2012 .

[63]  Masaki Kobayashi,et al.  Twisted quaternary neural networks , 2012 .

[64]  Akira Hirose,et al.  Complex-Valued Neural Networks , 2006, Studies in Computational Intelligence.

[65]  Nobuyuki Matsui,et al.  Quaternionic Multilayer Perceptron with Local Analyticity , 2012, Inf..

[66]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[67]  Frédéric Béchet,et al.  DECODA: a call-centre human-human spoken conversation corpus , 2012, LREC.

[68]  Eckhard Hitzer,et al.  Quaternion Fourier Transform on Quaternion Fields and Generalizations , 2007, ArXiv.

[69]  Sos S. Agaian,et al.  Quaternion Neural Networks Applied to Prostate Cancer Gleason Grading , 2013, 2013 IEEE International Conference on Systems, Man, and Cybernetics.

[70]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[71]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[72]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[73]  Kazuhiko Takahashi,et al.  Design of control systems using quaternion neural network and its application to inverse kinematics of robot manipulator , 2013, Proceedings of the 2013 IEEE/SICE International Symposium on System Integration.

[74]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[75]  Kazuhiko Takahashi,et al.  Remarks on Computational Facial Expression Recognition from HOG Features Using Quaternion Multi-layer Neural Network , 2014, EANN.

[76]  Dongpo Xu,et al.  Quaternion Derivatives: The GHR Calculus , 2014, 1409.8168.

[77]  Akira Hirose,et al.  Quaternion Neural-Network-Based PolSAR Land Classification in Poincare-Sphere-Parameter Space , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[78]  Masaki Kobayashi,et al.  Hybrid Quaternionic Hopfield Neural Network , 2015, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[79]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[80]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[81]  Dongpo Xu,et al.  Enabling quaternion derivatives: the generalized HR calculus , 2014, Royal Society Open Science.

[82]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[83]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[84]  Takehiko,et al.  Neural Network Inversion for Multilayer Quaternion Neural Networks , 2016 .

[85]  Titouan Parcollet,et al.  Quaternion Neural Networks for Spoken Language Understanding , 2016, 2016 IEEE Spoken Language Technology Workshop (SLT).

[86]  Titouan Parcollet,et al.  Deep quaternion neural networks for spoken language understanding , 2017, 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU).

[87]  Dongpo Xu,et al.  Learning Algorithms in Quaternion Neural Networks Using GHR Calculus , 2017 .

[88]  Calin-Adrian Popa.  Learning Algorithms for Quaternion-Valued Neural Networks , 2017, Neural Processing Letters.

[89]  Titouan Parcollet,et al.  Quaternion Denoising Encoder-Decoder for Theme Identification of Telephone Conversations , 2017, INTERSPEECH.

[90]  Kazuhiko Takahashi,et al.  Remarks on quaternion neural network-based controller trained by feedback error learning , 2017, 2017 IEEE/SICE International Symposium on System Integration (SII).

[91]  Igor N. Aizenberg,et al.  Image Recognition using MLMVN and Frequency Domain Features , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[92]  Ying Zhang,et al.  Quaternion Convolutional Neural Networks for End-to-End Automatic Speech Recognition , 2018, INTERSPEECH.

[93]  Eduardo Bayro-Corrochano,et al.  Geometric techniques for robotics and HMI: Interpolation and haptics in conformal geometric algebra and control using quaternion spike neural networks , 2018, Robotics Auton. Syst..

[94]  Marcos Eduardo Valle,et al.  On the Dynamics of Hopfield Neural Networks on Unit Quaternions , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[95]  Anthony S. Maida,et al.  Deep Quaternion Networks , 2017, 2018 International Joint Conference on Neural Networks (IJCNN).

[96]  Akira Hirose,et al.  Isotropization of Quaternion-Neural-Network-Based PolSAR Adaptive Land Classification in Poincare-Sphere Parameter Space , 2018, IEEE Geoscience and Remote Sensing Letters.

[97]  Danilo Comminiello,et al.  Quaternion Convolutional Neural Networks for Detection and Localization of 3D Sound Events , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[98]  Archontis Politis,et al.  Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks , 2018, IEEE Journal of Selected Topics in Signal Processing.

[99]  Titouan Parcollet,et al.  Quaternion Convolutional Neural Networks for Heterogeneous Image Processing , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).