On the Learning Machine with Amplificatory Neuron in Complex Domain

The processing of complex-valued signals through neural networks is the important and challenging fields in image processing and digital signal processing. The networks with linear neurons in complex domain have been proposed in the various literatures, but its convergence capability and computational power are not up the mark. In the various researches, it has been mentioned that nonlinear aggregation of input signals provides better computational capability as compared to linear aggregation. These evidences motivated to design a new nonlinear aggregation operation. In this paper, a new artificial neuron structure based on nonlinear aggregation of complex-valued input signals is proposed. The learning rule of network with proposed neurons is also addressed to achieve faster learning and better computational power. Its aggregation operation is based on the product of different weighted arrangement of inputs with bias signals instead of summation. This product exhibits nonlinearity that amplifies the aggregation operation of the proposed new neuron and named as AMPlificatory neuron in Complex domain ( $${\mathbb {C}}$$ C -AMP). The training and testing processes of the three-layered network with $${\mathbb {C}}$$ C -AMP neurons are performed through standard classification, prediction, and function approximation problems for evaluating the computational capability of the proposed neuron and compared with existing neuron models. Results of training and testing processes for these problems through proposed work ensure better training, faster convergence, excellent generalization ability, and significant prediction accuracy with lesser network topology as compared to conventional neuron models. The excellent generalization for 2D transformation problems also shows the significant capability of $${\mathbb {C}}$$ C -AMP neuron.

[1]  D. Mandic,et al.  Complex Valued Nonlinear Adaptive Filters: Noncircularity, Widely Linear and Neural Models , 2009 .

[2]  G. Laurent,et al.  Computation of Object Approach by a Wide-Field, Motion-Sensitive Neuron , 1999, The Journal of Neuroscience.

[3]  Snehashish Chakraverty,et al.  McCulloch–Pitts Neural Network Model , 2019, Concepts of Soft Computing.

[4]  Rui Liu,et al.  Effective long short-term memory with differential evolution algorithm for electricity price prediction , 2018, Energy.

[5]  Madan M. Gupta,et al.  Static and Dynamic Neural Networks: From Fundamentals to Advanced Theory , 2003 .

[6]  Martin T. Hagan,et al.  Backpropagation Algorithms for a Broad Class of Dynamic Networks , 2007, IEEE Transactions on Neural Networks.

[7]  Prem Kumar Kalra,et al.  The novel aggregation function-based neuron models in complex domain , 2010, Soft Comput..

[8]  Jacek M. Zurada,et al.  Blur Identification by Multilayer Neural Network Based on Multivalued Neurons , 2008, IEEE Transactions on Neural Networks.

[9]  Necati Özdemir,et al.  Complex valued neural network with Möbius activation function , 2011 .

[10]  Konstantin Eckle,et al.  A comparison of deep networks with ReLU activation function and linear spline-type methods , 2018, Neural Networks.

[11]  Gwilym M. Jenkins,et al.  Time series analysis, forecasting and control , 1971 .

[12]  Boris Hanin,et al.  Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations , 2017, Mathematics.

[13]  Christof Koch,et al.  Biophysics of Computation: Information Processing in Single Neurons (Computational Neuroscience Series) , 1998 .

[14]  Danilo P. Mandic,et al.  Complex Valued Nonlinear Adaptive Filters , 2009 .

[15]  Yasuaki Kuroe,et al.  On Activation Functions for Complex-Valued Neural Networks - Existence of Energy Functions , 2003, ICANN.

[16]  Igor N. Aizenberg,et al.  Complex-Valued Neural Networks with Multi-Valued Neurons , 2011, Studies in Computational Intelligence.

[17]  Ramazan Coban A context layered locally recurrent neural network for dynamic system identification , 2013, Eng. Appl. Artif. Intell..

[18]  Prem Kumar Kalra,et al.  Functional mapping with complex higher order compensatory neuron model , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[19]  Haipeng Wang,et al.  Complex-Valued Convolutional Neural Network and Its Application in Polarimetric SAR Image Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[20]  Abir Jaafar Hussain,et al.  The application of ridge polynomial neural network to multi-step ahead financial time series prediction , 2008, Neural Computing and Applications.

[21]  Qingyu Xiong,et al.  Two-Stage Method for Diagonal Recurrent Neural Network Identification of a High-Power Continuous Microwave Heating System , 2019, Neural Processing Letters.

[22]  Tohru Nitta,et al.  An Extension of the Back-Propagation Algorithm to Complex Numbers , 1997, Neural Networks.

[23]  Danilo P. Mandic,et al.  An Augmented Echo State Network for Nonlinear Adaptive Filtering of Complex Noncircular Signals , 2011, IEEE Transactions on Neural Networks.

[24]  N Suga,et al.  Specialized subsystems for processing biologically important complex sounds: cross-correlation analysis for ranging in the bat's brain. , 1990, Cold Spring Harbor symposia on quantitative biology.

[25]  Nicholas J. Redding,et al.  Constructive higher-order network that is polynomial time , 1993, Neural Networks.

[26]  Nirmal K. Bose,et al.  Landmine detection and classification with complex-valued hybrid neural network using scattering parameters dataset , 2005, IEEE Transactions on Neural Networks.

[27]  Akira Watanabe,et al.  A Method to Interpret 3D Motions Using Neural Networks (Special Section on Information Theory and Its Applications) , 1994 .

[28]  Devendra K. Chaturvedi,et al.  New neuron models for simulating rotating electrical machines and load forecasting problems , 1999 .

[29]  Tohru Nitta An Analysis of the Fundamental Structure of Complex-Valued Neurons , 2004, Neural Processing Letters.

[30]  Xiaoming Chen,et al.  A modified error backpropagation algorithm for complex-value neural networks , 2005, Int. J. Neural Syst..

[31]  Francesco Piazza,et al.  On the complex backpropagation algorithm , 1992, IEEE Trans. Signal Process..

[32]  Akira Hirose Complex-Valued Neural Networks , 2006, Studies in Computational Intelligence.

[33]  Andrei D. Polyanin,et al.  Handbook of Linear Partial Differential Equations for Engineers and Scientists, Second Edition , 2016 .

[34]  Wai Lok Woo,et al.  Weierstrass approach to blind source separation of multiple nonlinearly mixed signals , 2006 .

[35]  Prem Kumar Kalra,et al.  On Efficient Learning Machine With Root-Power Mean Neuron in Complex Domain , 2011, IEEE Transactions on Neural Networks.

[36]  John Y. Cheung,et al.  Polynomial and standard higher order neural network , 1993, IEEE International Conference on Neural Networks.

[37]  Olaf Hellwich,et al.  Complex-Valued Convolutional Neural Networks for Object Detection in PolSAR data , 2010 .

[38]  Eitaro Aiyoshi,et al.  Approximation and Designing of Fractal Images by Complex Neural Networks , 2003 .

[39]  T. Poggio,et al.  On optimal nonlinear associative recall , 1975, Biological Cybernetics.

[40]  Tohru Nitta,et al.  Orthogonality of Decision Boundaries in Complex-Valued Neural Networks , 2004, Neural Computation.

[41]  Yuanzhi Li,et al.  Convergence Analysis of Two-layer Neural Networks with ReLU Activation , 2017, NIPS.

[42]  Lin Wang,et al.  Effective electricity energy consumption forecasting using echo state network improved by differential evolution algorithm , 2018 .

[43]  Liang Ding,et al.  Diagonal recurrent neural networks for parameters identification of terrain based on wheel–soil interaction analysis , 2017, Neural Computing and Applications.

[44]  Bin Gao,et al.  Single-channel blind separation using L₁-sparse complex non-negative matrix factorization for acoustic signals. , 2015, The Journal of the Acoustical Society of America.

[45]  A. Polyanin Handbook of Linear Partial Differential Equations for Engineers and Scientists , 2001 .

[46]  Claudio Moraga,et al.  Multilayer Feedforward Neural Network Based on Multi-valued Neurons (MLMVN) and a Backpropagation Learning Algorithm , 2006, Soft Comput..

[47]  Henry Leung,et al.  The complex backpropagation algorithm , 1991, IEEE Trans. Signal Process..

[48]  Bartlett W. Mel,et al.  Information Processing in Dendritic Trees , 1994, Neural Computation.

[49]  Shan Liu,et al.  Optimal Forecast Combination Based on Neural Networks for Time Series Forecasting , 2018, Appl. Soft Comput..

[50]  I. Ohzawa,et al.  Neural mechanisms for processing binocular information I. Simple cells. , 1999, Journal of neurophysiology.

[51]  Donq-Liang Lee Improving the capacity of complex-valued neural networks with a modified gradient descent learning rule , 2001, IEEE Trans. Neural Networks.

[52]  R. M. Siegel,et al.  Encoding of spatial location by posterior parietal neurons. , 1985, Science.

[53]  Sushil Kumar,et al.  Root-Power Mean Aggregation-Based Neuron in Quaternionic Domain , 2019 .

[54]  Mohammad Hassan Moradi,et al.  A Hybrid Higher Order Neural Classifier for handling classification problems , 2011, Expert Syst. Appl..

[55]  G. Laurent,et al.  Elementary Computation of Object Approach by a Wide-Field Visual Neuron , 1995, Science.

[56]  Tülay Adali,et al.  Approximation by Fully Complex Multilayer Perceptrons , 2003, Neural Computation.

[57]  Kwang Y. Lee,et al.  Diagonal recurrent neural networks for dynamic systems control , 1995, IEEE Trans. Neural Networks.

[58]  Yüksel Özbay,et al.  Fuzzy clustering complex-valued neural network to diagnose cirrhosis disease , 2011, Expert Syst. Appl..

[59]  Gwilym M. Jenkins,et al.  Time series analysis, forecasting and control , 1972 .

[60]  Prem Kumar Kalra,et al.  Time series prediction with single multiplicative neuron model , 2007, Appl. Soft Comput..

[61]  Narasimhan Sundararajan,et al.  Fully complex extreme learning machine , 2005, Neurocomputing.

[62]  Michael W. Spratling,et al.  Learning Synaptic Clusters for Nonlinear Dendritic Processing , 2004, Neural Processing Letters.

[63]  Charles R. Johnson,et al.  Topics in matrix analysis: The Hadamard product , 1991 .

[64]  J. W. Brown,et al.  Complex Variables and Applications , 1985 .

[65]  Qiankun Song,et al.  Boundedness and Complete Stability of Complex-Valued Neural Networks With Time Delay , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[66]  Bipin Kumar Tripathi Higher-Order Computational Model for Novel Neurons , 2015 .

[67]  T. Poggio,et al.  Multiplying with synapses and neurons , 1992 .

[68]  Kazuyuki Murase,et al.  Single-layered complex-valued neural network for real-valued classification problems , 2009, Neurocomputing.

[69]  N. Sundararajan,et al.  A fully complex-valued radial basis function network and its learning algorithm. , 2009 .

[70]  Paolo Campolucci,et al.  Complex-valued neural networks with adaptive spline activation function for digital-radio-links nonlinear equalization , 1999, IEEE Trans. Signal Process..

[71]  Cris Koutsougeras,et al.  Complex domain backpropagation , 1992 .

[72]  Bipin Kumar Tripathi,et al.  On the complex domain deep machine learning for face recognition , 2017, Applied Intelligence.

[73]  Naotake Kamiura,et al.  Associative Memory in quaternionic Hopfield Neural Network , 2008, Int. J. Neural Syst..

[74]  Kotaro Hirasawa,et al.  Multiplication units in feedforward neural networks and its training , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[75]  Christof Koch,et al.  The role of single neurons in information processing , 2000, Nature Neuroscience.

[76]  W. L. Woo,et al.  Non-linear independent component analysis using series reversion and Weierstrass network , 2006 .