Feed forward neural network with random quaternionic neurons

A quaternionic extension of feed forward neural network, for processing multi-dimensional signals, is proposed in this paper. This neural network is based on the three layered network with random weights, called Extreme Learning Machines (ELMs), in which iterative least-mean-square algorithms are not required for training networks. All parameters and variables in the proposed network are encoded by quaternions and operations among them follow the quaternion algebra. Neurons in the proposed network are expected to operate multi-dimensional signals as single entities, rather than real-valued neurons deal with each element of signals independently. The performances for the proposed network are evaluated through two types of experiments: classifications and reconstructions for color images in the CIFAR-10 dataset. The experimental results show that the proposed networks are superior in terms of classification accuracies for input images than the conventional (real-valued) networks with similar degrees of freedom. The detailed investigations for operations in the proposed networks are conducted. HighlightsA feedforward neural network for accepting three- or four-dimension is proposed.Quaternions, which are a four-dimensional hypercomlex numbers, are used for encoding neuronal parameters.Its performances are evaluated through the classification and autoencoding for CIFAR-10 dataset.

[1]  Fuzhen Zhang.  Quaternions and matrices of quaternions , 1997 .

[2]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[3]  Guang-Bin Huang,et al.  Extreme Learning Machine for Multilayer Perceptron , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[4]  B. Schrauwen,et al.  Reservoir computing and extreme learning machines for non-linear time-series data analysis , 2013, Neural Networks.

[5]  Akira Hirose,et al.  Complex-Valued Neural Networks: Theories and Applications , 2003 .

[6]  Eckhard Hitzer,et al.  Quaternion Fourier Transform on Quaternion Fields and Generalizations , 2007, ArXiv.

[7]  C. Schwartz.  Calculus with a quaternionic variable , 2008, 0812.1185.

[8]  Hongming Zhou,et al.  Representational Learning with ELMs for Big Data , 2013 .

[9]  P. Arena,et al.  Quaternionic Multilayer Perceptrons for Chaotic Time Series Prediction , 1996 .

[10]  Thomas L. Hankins,et al.  Sir William Rowan Hamilton , 1980 .

[11]  T. Nitta,et al.  A Solution to the 4-bit Parity Problem with a Single Quaternary Neuron , 2004 .

[12]  Herbert Jaeger,et al.  Adaptive Nonlinear System Identification with Echo State Networks , 2002, NIPS.

[13]  Nobuyuki Matsui,et al.  Quaternionic Multilayer Perceptron with Local Analyticity , 2012, Inf..

[14]  Danilo P. Mandic,et al.  Split quaternion nonlinear adaptive filtering , 2010, Neural Networks.

[15]  Danilo P. Mandic,et al.  Quaternion-Valued Nonlinear Adaptive Filtering , 2011, IEEE Transactions on Neural Networks.

[16]  William Rowan Hamilton,et al.  Lectures on quaternions , 1853 .

[17]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[18]  Stephen J. Sangwine,et al.  Quaternion involutions and anti-involutions , 2005, Comput. Math. Appl..

[19]  Guang-Bin Huang,et al.  Trends in extreme learning machines: A review , 2015, Neural Networks.

[20]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[21]  S. G. Hoggar.  Mathematics for computer graphics , 1993, Cambridge tracts in theoretical computer science.

[22]  Danilo P. Mandic,et al.  A Quaternion Gradient Operator and Its Applications , 2011, IEEE Signal Processing Letters.

[23]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[24]  Nicolas Le Bihan,et al.  Polarized Signal Classification by Complex and quaternionic Multi-Layer Perceptrons , 2008, Int. J. Neural Syst..

[25]  Victor C. M. Leung,et al.  Extreme Learning Machines [Trends & Controversies] , 2013, IEEE Intelligent Systems.

[26]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[27]  Nobuyuki Matsui,et al.  Quaternion neural network with geometrical operators , 2004, J. Intell. Fuzzy Syst..

[28]  Hiromi Kusamichi,et al.  A New Scheme for Color Night Vision by Quaternion Neural Network , 2004 .

[29]  Dipankar Das,et al.  Enhanced SenticNet with Affective Labels for Concept-Based Opinion Mining , 2013, IEEE Intelligent Systems.

[30]  Danilo P. Mandic,et al.  Quaternion-Valued Echo State Networks , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[31]  Jack B. Kuipers,et al.  Quaternions and Rotation Sequences: A Primer with Applications to Orbits, Aerospace and Virtual Reality , 2002 .

[32]  J. L. Brenner,et al.  Matrices of quaternions. , 1951 .

[33]  Chi-Man Vong,et al.  Local Receptive Fields Based Extreme Learning Machine , 2015, IEEE Computational Intelligence Magazine.

[34]  Danilo P. Mandic,et al.  Kalman filtering for widely linear complex and quaternion valued bearings only tracking , 2012, IET Signal Process..

[35]  Luigi Fortuna,et al.  Chaotic time series prediction via quaternionic multilayer perceptrons , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.

[36]  Danilo P. Mandic,et al.  The Theory of Quaternion Matrix Derivatives , 2014, IEEE Transactions on Signal Processing.

[37]  Stephen J. Sangwine,et al.  Hypercomplex Fourier Transforms of Color Images , 2007, IEEE Trans. Image Process..

[38]  T. Nitta.  An Extension of the Back-propagation Algorithm to Quaternions , 1996 .

[39]  Giovanni Muscato,et al.  Multilayer Perceptrons to Approximate Quaternion Valued Functions , 1997, Neural Networks.

[40]  Narasimhan Sundararajan,et al.  Fully complex extreme learning machine , 2005, Neurocomputing.