A recurrent log-linearized Gaussian mixture network

Context in time series is one of the most useful and interesting characteristics for machine learning. In some cases, the dynamic characteristic would be the only basis for achieving a possible classification. A novel neural network, which is named "a recurrent log-linearized Gaussian mixture network (R-LLGMN)," is proposed in this paper for classification of time series. The structure of this network is based on a hidden Markov model (HMM), which has been well developed in the area of speech recognition. R-LLGMN can as well be interpreted as an extension of a probabilistic neural network using a log-linearized Gaussian mixture model, in which recurrent connections have been incorporated to make temporal information in use. Some simulation experiments are carried out to compare R-LLGMN with the traditional estimator of HMM as classifiers, and finally, pattern classification experiments for EEG signals are conducted. It is indicated from these experiments that R-LLGMN can successfully classify not only artificial data but real biological data such as EEG signals.

[1]  Christopher M. Bishop,et al.  Neural Network for Pattern Recognition , 1995 .

[2]  Roy L. Streit,et al.  Maximum likelihood training of probabilistic neural networks , 1994, IEEE Trans. Neural Networks.

[3]  John Scott Bridle,et al.  Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition , 1989, NATO Neurocomputing.

[4]  Donald F. Specht,et al.  Probabilistic neural networks , 1990, Neural Networks.

[5]  L. Baum,et al.  Statistical Inference for Probabilistic Functions of Finite State Markov Chains , 1966 .

[6]  Anthony J. Robinson,et al.  An application of recurrent nets to phone probability estimation , 1994, IEEE Trans. Neural Networks.

[7]  L. Baum,et al.  A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains , 1970 .

[8]  Richard Lippmann,et al.  Neural Network Classifiers Estimate Bayesian a posteriori Probabilities , 1991, Neural Computation.

[9]  C. Lee Giles,et al.  How embedded memory in recurrent neural network architectures helps learning long-term temporal dependencies , 1998, Neural Networks.

[10]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[11]  Horacio Franco,et al.  Hybrid neural network/hidden Markov model continuous-speech recognition , 1992, ICSLP.

[12]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[13]  Georg Dorffner,et al.  Fat tails and non-linearity in volatility models: what is more important? , 1999, Proceedings of the IEEE/IAFE 1999 Conference on Computational Intelligence for Financial Engineering (CIFEr) (IEEE Cat. No.99TH8408).

[14]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[15]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[16]  John S. Bridle,et al.  Alpha-nets: A recurrent 'neural' network architecture with a hidden Markov model interpretation , 1990, Speech Commun..

[17]  Toshio Tsuji,et al.  Pattern classification of time-series EMG signals using neural networks , 2000 .

[18]  Donald C. Wunsch,et al.  Recurrent neural network based prediction of epileptic seizures in intra- and extracranial EEG , 2000, Neurocomputing.

[19]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[20]  Nikolaos G. Bourbakis,et al.  Handwritten character recognition using low resolutions , 1999 .

[21]  Jie Zhang,et al.  Long-term prediction models based on mixed order locally recurrent neural networks , 1998 .

[22]  Mary P. Harper,et al.  Stochastic observation hidden Markov models , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[23]  Marco Saerens,et al.  Hopfield net generation, encoding and classification of temporal trajectories , 1994, IEEE Trans. Neural Networks.

[24]  Toshio Tsuji,et al.  A log-linearized Gaussian mixture network and its application to EEG pattern classification , 1999, IEEE Trans. Syst. Man Cybern. Part C.

[25]  Leonid I. Perlovsky,et al.  Maximum likelihood neural networks for sensor fusion and adaptive classification , 1991, Neural Networks.

[26]  M. Zak.  Terminal attractors for addressable memory in neural networks , 1988 .

[27]  Michael I. Jordan.  Serial Order: A Parallel Distributed Processing Approach , 1997 .

[28]  Terry Caelli,et al.  Model-based neural networks , 1993, Neural Networks.

[29]  Nikko Strom,et al.  Phoneme probability estimation with dynamic sparsely connected artificial neural networks , 1997 .

[30]  H. Bourlard,et al.  Links Between Markov Models and Multilayer Perceptrons , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Biing-Hwang Juang,et al.  Maximum likelihood estimation for multivariate mixture observations of markov chains , 1986, IEEE Trans. Inf. Theory.

[32]  Toshio Tsuji,et al.  EMG-based human-robot interface for rehabilitation aid , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[33]  Lawrence R. Rabiner,et al.  A segmental k-means training procedure for connected word recognition , 1986, AT&T Technical Journal.

[34]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[35]  Toshio Tsuji,et al.  Motion Discrimination Method from EMG Signals Using Statistically Structured Neural Networks , 1992 .

[36]  Hans G. C. Tråvén,et al.  A neural network approach to statistical pattern classification by 'semiparametric' estimation of probability density functions , 1991, IEEE Trans. Neural Networks.

[37]  Alex Aussem,et al.  Dynamical recurrent neural networks towards prediction and modeling of dynamical systems , 1999, Neurocomputing.

[38]  S. Shimoji,et al.  Self-organization of Gaussian mixture model for learning class pdfs in pattern classification , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[39]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.