An Augmented Echo State Network for Nonlinear Adaptive Filtering of Complex Noncircular Signals

A novel complex echo state network (ESN), utilizing full second-order statistical information in the complex domain, is introduced. This is achieved through the use of the so-called augmented complex statistics, thus making complex ESNs suitable for processing the generality of complex-valued signals, both second-order circular (proper) and noncircular (improper). Next, in order to deal with nonstationary processes with large nonlinear dynamics, a nonlinear readout layer is introduced and is further equipped with an adaptive amplitude of the nonlinearity. This combination of augmented complex statistics and enhanced adaptivity within ESNs also facilitates the processing of bivariate signals with strong component correlations. Simulations in the prediction setting on both circular and noncircular synthetic benchmark processes and real-world noncircular and nonstationary wind signals support the analysis.

[1]  Danilo P. Mandic,et al.  THE AUGMENTED COMPLEX LEAST MEAN SQUARE ALGORITHM WITH APPLICATION TO ADAPTIVE PREDICTION PROBLEMS 1 , 2008 .

[2]  Cris Koutsougeras,et al.  Complex domain backpropagation , 1992 .

[3]  Peter J. Schreier.  The degree of impropriety (noncircularity) of complex random vectors , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[4]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[5]  Herbert Jaeger,et al.  A tutorial on training recurrent neural networks , covering BPPT , RTRL , EKF and the " echo state network " approach - Semantic Scholar , 2005 .

[6]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[7]  Daniel S. Yeung,et al.  Adaptive Filter Design Using Recurrent Cerebellar Model Articulation Controller , 2010, IEEE Transactions on Neural Networks.

[8]  Joos Vandewalle,et al.  Multi-Valued and Universal Binary Neurons: Theory, Learning and Applications , 2012 .

[9]  Tohru Nitta,et al.  An Extension of the Back-Propagation Algorithm to Complex Numbers , 1997, Neural Networks.

[10]  James L. Massey,et al.  Proper complex random processes with applications to information theory , 1993, IEEE Trans. Inf. Theory.

[11]  Danilo P. Mandic,et al.  An augmented CRTRL for complex-valued recurrent neural networks , 2007, Neural Networks.

[12]  Danilo P. Mandic,et al.  Complex Valued Nonlinear Adaptive Filters , 2009 .

[13]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[14]  B. A. D. H. Brandwood.  A complex gradient operator and its applica-tion in adaptive array theory , 1983 .

[15]  Jacek M. Zurada,et al.  Complex-valued multistate neural associative memory , 1996, IEEE Trans. Neural Networks.

[16]  Ching-Hung Lee,et al.  Identification and control of dynamic systems using recurrent fuzzy neural networks , 2000, IEEE Trans. Fuzzy Syst..

[17]  Richard D. Braatz,et al.  On the "Identification and control of dynamical systems using neural networks" , 1997, IEEE Trans. Neural Networks.

[18]  Jun Wang,et al.  Information theoretic learning applied to wind power modeling , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[19]  J. Navarro-Moreno,et al.  ARMA Prediction of Widely Linear Systems by Using the Innovations Algorithm , 2008, IEEE Transactions on Signal Processing.

[20]  Edmondo Trentin,et al.  Networks with trainable amplitude of activation functions , 2001, Neural Networks.

[21]  Joos Vandewalle,et al.  Multi-Valued and Universal Binary Neurons , 2000 .

[22]  N.D. Hatziargyriou,et al.  An Advanced Statistical Method for Wind Power Forecasting , 2007, IEEE Transactions on Power Systems.

[23]  Stephen A. Billings,et al.  Recurrent radial basis function networks for adaptive noise cancellation , 1995, Neural Networks.

[24]  Louis L. Scharf,et al.  Second-order analysis of improper complex random vectors and processes , 2003, IEEE Trans. Signal Process..

[25]  W. Wirtinger.  Zur formalen Theorie der Funktionen von mehr komplexen Veränderlichen , 1927 .

[26]  Akira Hirose,et al.  Complex-Valued Neural Networks , 2006, Studies in Computational Intelligence.

[27]  Weifeng Liu,et al.  An Information Theoretic Approach of Designing Sparse Kernel Adaptive Filters , 2009, IEEE Transactions on Neural Networks.

[28]  Danilo P. Mandic,et al.  Toward an optimal PRNN-based nonlinear predictor , 1999, IEEE Trans. Neural Networks.

[29]  D. Mandic,et al.  Complex Valued Nonlinear Adaptive Filters: Noncircularity, Widely Linear and Neural Models , 2009 .

[30]  Xiaohong Jiang,et al.  Generalized Two-Hop Relay for Flexible Delay Control in MANETs , 2012, IEEE/ACM Transactions on Networking.

[31]  A. Bos.  Complex gradient and Hessian , 1994 .

[32]  J.C. Principe,et al.  Signal Processing with Echo State Networks in the Complex Domain , 2007, 2007 IEEE Workshop on Machine Learning for Signal Processing.

[33]  Liang Li,et al.  Nonlinear adaptive prediction of nonstationary signals , 1995, IEEE Trans. Signal Process..

[34]  Danilo P. Mandic,et al.  Adaptive IIR Filtering of Noncircular Complex Signals , 2009, IEEE Transactions on Signal Processing.

[35]  Ken Kreutz-Delgado,et al.  The Complex Gradient Operator and the CR-Calculus ECE275A - Lecture Supplement - Fall 2005 , 2009, 0906.4835.

[36]  Pascal Chevalier,et al.  Widely linear estimation with complex data , 1995, IEEE Trans. Signal Process..

[37]  B. Widrow,et al.  The complex LMS algorithm , 1975, Proceedings of the IEEE.

[38]  Danilo P. Mandic,et al.  An Augmented Extended Kalman Filter Algorithm for Complex-Valued Recurrent Neural Networks , 2006, Neural Computation.

[39]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[40]  Kazuyuki Aihara,et al.  Complex-valued prediction of wind profile using augmented complex statistics , 2009 .

[41]  Yili Xia,et al.  A complex Echo State Network for nonlinear adaptive filtering , 2008, 2008 IEEE Workshop on Machine Learning for Signal Processing.

[42]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[43]  L. Scharf,et al.  Statistical Signal Processing of Complex-Valued Data: The Theory of Improper and Noncircular Signals , 2010 .