Neural Networks and Nonlinear Adaptive Filtering: Unifying Concepts and New Algorithms

The paper proposes a general framework that encompasses the training of neural networks and the adaptation of filters. We show that neural networks can be considered as general nonlinear filters that can be trained adaptively, that is, that can undergo continual training with a possibly infinite number of time-ordered examples. We introduce the canonical form of a neural network. This canonical form permits a unified presentation of network architectures and of gradient-based training algorithms for both feedforward networks (transversal filters) and feedback networks (recursive filters). We show that several algorithms used classically in linear adaptive filtering, and some algorithms suggested by other authors for training neural networks, are special cases in a general classification of training algorithms for feedback networks.

[1]  S. Applebaum,et al.  Adaptive arrays with main beam constraints , 1976 .

[2]  S. Applebaum,et al.  Adaptive arrays , 1976 .

[3]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[4]  Gary L. Wise,et al.  On the design of nonlinear discrete-time predictors , 1982, IEEE Trans. Inf. Theory.

[5]  John G. Proakis,et al.  Digital Communications , 1983 .

[6]  Bernard Widrow,et al.  Adaptive Signal Processing , 1985 .

[7]  K. H. Barratt.  Digital Coding of Waveforms , 1985 .

[8]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[9]  Peter No,et al.  Digital Coding of Waveforms , 1986 .

[10]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[11]  L. Personnaz,et al.  Collective computational properties of neural networks: New learning mechanisms. , 1986, Physical review. A, General physics.

[12]  Robert M. Farber,et al.  How Neural Nets Work , 1987, NIPS.

[13]  Pineda,et al.  Generalization of back-propagation to recurrent neural networks. , 1987, Physical review letters.

[14]  Maurice Bellanger,et al.  Adaptive digital filters and signal analysis , 1987 .

[15]  Fernando J. Pineda,et al.  GENERALIZATION OF BACKPROPAGATION TO RECURRENT AND HIGH-ORDER NETWORKS. , 1987 .

[16]  Barak A. Pearlmutter.  Learning State Space Trajectories in Recurrent Neural Networks , 1989, Neural Computation.

[17]  Ronald J. Williams,et al.  Experimental Analysis of the Real-time Recurrent Learning Algorithm , 1989 .

[18]  B. Picinbono,et al.  Adaptive Methods in Temporal Processing , 1989 .

[19]  Frank Fallside,et al.  Analysis of linear predictive data as speech and of ARMA processes by a class of single-layer connectionist models , 1989, NATO Neurocomputing.

[20]  Fernando J. Pineda,et al.  Recurrent Backpropagation and the Dynamical Approach to Adaptive Neural Computation , 1989, Neural Computation.

[21]  Geoffrey E. Hinton,et al.  Phoneme recognition using time-delay neural networks , 1989, IEEE Trans. Acoust. Speech Signal Process..

[22]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[23]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[24]  Sheng Chen,et al.  Representations of non-linear systems: the NARMAX model , 1989 .

[25]  J. Shynk.  Adaptive IIR filtering , 1989, IEEE ASSP Magazine.

[26]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[27]  N. Jayant,et al.  Digital Coding of Waveforms: Principles and Applications to Speech and Video , 1990 .

[28]  K S Narendra,et al.  IDENTIFICATION AND CONTROL OF DYNAMIC SYSTEMS USING NEURAL NETWORKS , 1990 .

[29]  Jing Peng,et al.  An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories , 1990, Neural Computation.

[30]  David E. Rumelhart,et al.  Predicting the Future: a Connectionist Approach , 1990, Int. J. Neural Syst..

[31]  William H. Press,et al.  Numerical recipes , 1990 .

[32]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[33]  C.F.N. Cowan,et al.  Adaptive equalization of finite nonlinear channels using multilayer perceptron , 1990 .

[34]  K. P. Unnikrishnan,et al.  Nonlinear prediction of speech signals using memory neuron networks , 1991, Neural Networks for Signal Processing Proceedings of the 1991 IEEE Workshop.

[35]  Kumpati S. Narendra,et al.  Gradient methods for the optimization of dynamical systems containing neural networks , 1991, IEEE Trans. Neural Networks.

[36]  L. Personnaz,et al.  Adaptive training of feedback neural networks for non-linear filtering , 1992, Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop.