Enhanced Training for the Locally Recurrent Probabilistic Neural Networks

In the present contribution we propose an integral training procedure for the Locally Recurrent Probabilistic Neural Networks (LR PNNs). Specifically, the adjustment of the smoothing factor "sigma" in the pattern layer of the LR PNN and the training of the recurrent layer weights are integrated in an automatic process that iteratively estimates all adjustable parameters of the LR PNN from the available training data. Furthermore, in contrast to the original LR PNN, whose recurrent layer was trained to provide optimum separation among the classes on the training dataset, while striving to keep a balance between the learning rates for all classes, here the training strategy is oriented towards optimizing the overall classification accuracy, straightforwardly. More precisely, the new training strategy directly targets at maximizing the posterior probabilities for the target class and minimizing the posterior probabilities estimated for the non-target classes. The new fitness function requires fewer computations for each evaluation, and therefore the overall computational demands for training the recurrent layer weights are reduced. The performance of the integrated training procedure is illustrated on three different speech processing tasks: emotion recognition, speaker identification and speaker verification.

[1]  Herbert Reininger,et al.  Strategies for reducing the complexity of a RNN based speech recognizer , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[2]  Jing J. Liang,et al.  Comprehensive learning particle swarm optimizer for global optimization of multimodal functions , 2006, IEEE Transactions on Evolutionary Computation.

[3]  Pietro Burrascano,et al.  Learning vector quantization for the probabilistic neural network , 1991, IEEE Trans. Neural Networks.

[4]  Nicos G. Pavlidis,et al.  New Self-adaptive Probabilistic Neural Networks in Bioinformatic and Medical Tasks , 2006, Int. J. Artif. Intell. Tools.

[5]  Enrico Zio,et al.  Simulating the dynamics of the neutron flux in a nuclear reactor by locally recurrent neural networks , 2007 .

[6]  Roy L. Streit,et al.  Maximum likelihood training of probabilistic neural networks , 1994, IEEE Trans. Neural Networks.

[7]  Sun-Yuan Kung,et al.  Face recognition/detection by probabilistic decision-based neural network , 1997, IEEE Trans. Neural Networks.

[8]  Michael N. Vrahatis,et al.  Optimal power allocation and joint source-channel coding for wireless DS-CDMA visual sensor networks using the Nash Bargaining Solution , 2005, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[9]  Shaogang Gong,et al.  Modelling spatio-temporal trajectories and face signatures on partially recurrent neural networks , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[10]  R. Storn,et al.  Differential Evolution - A simple and efficient adaptive scheme for global optimization over continuous spaces , 2004 .

[11]  Aurelio Uncini,et al.  Sound Synthesis by Flexible Activation Function Recurrent Neural Networks , 2002, WIRN.

[12]  Dimitris K. Tasoulis,et al.  Locally recurrent probabilistic neural network for text-independent speaker verification , 2003, INTERSPEECH.

[13]  Sun-Yuan Kung,et al.  Estimation of elliptical basis function parameters by the EM algorithm with application to speaker verification , 2000, IEEE Trans. Neural Networks Learn. Syst..

[14]  A. C. Tsoi,et al.  Nonlinear system identification using multilayer perceptrons with locally recurrent synaptic structure , 1992, Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop.

[15]  Ioannis B. Theocharis,et al.  Locally recurrent neural networks for wind speed prediction using spatial correlation , 2007, Inf. Sci..

[16]  Donald F. Specht,et al.  Probabilistic neural networks , 1990, Neural Networks.

[17]  Michele Marchesi,et al.  Neural networks for power system condition monitoring and protection , 1998, Neurocomputing.

[18]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[19]  Lihua Xie,et al.  Active control of sound based on diagonal recurrent neural network , 2002, Proceedings of the 41st SICE Annual Conference. SICE 2002..

[20]  Kwang Y. Lee,et al.  Diagonal recurrent neural networks for dynamic systems control , 1995, IEEE Trans. Neural Networks.

[21]  Anton Batliner,et al.  Speaker Characteristics and Emotion Classification , 2007, Speaker Classification.

[22]  Eric Sanders,et al.  Speechdat multilingual speech databases for teleservices: across the finish line , 1999, EUROSPEECH.

[23]  Ioan Cristian Trelea,et al.  The particle swarm optimization algorithm: convergence analysis and parameter selection , 2003, Inf. Process. Lett..

[24]  Sheng Chen,et al.  Robust maximum likelihood training of heteroscedastic probabilistic neural networks , 1998, Neural Networks.

[25]  Joseph Bibb Cain.  Improved probabilistic neural network and its performance relative to other models , 1990, Defense, Security, and Sensing.

[26]  J. A. Hartigan,et al.  A k-means clustering algorithm , 1979 .

[27]  Herbert Reininger,et al.  A SPEECH RECOGNIZER BASED ON LOCALLY RECURRENT NEURAL NETWORKS , 1995 .

[28]  Dominique Genoud,et al.  POLYCOST: A telephone-speech database for speaker recognition , 2000, Speech Commun..

[29]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[30]  Fionn Murtagh,et al.  Dynamical recurrent neural networks -- towards environmental time series prediction , 1995, Int. J. Neural Syst..

[31]  Michael R. Berthold,et al.  Constructive training of probabilistic neural networks , 1998, Neurocomputing.

[32]  Ioannis B. Theocharis,et al.  A locally recurrent fuzzy neural network with application to the wind speed prediction using spatial correlation , 2007, Neurocomputing.

[33]  C. Lee Giles,et al.  How embedded memory in recurrent neural network architectures helps learning long-term temporal dependencies , 1998, Neural Networks.

[34]  Ah Chung Tsoi,et al.  FIR and IIR Synapses, a New Neural Network Architecture for Time Series Modeling , 1991, Neural Computation.

[35]  Martin T. Hagan,et al.  Gauss-Newton approximation to Bayesian learning , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[36]  Maurice Clerc,et al.  The particle swarm - explosion, stability, and convergence in a multidimensional complex space , 2002, IEEE Trans. Evol. Comput..

[37]  Dimitris K. Tasoulis,et al.  Generalized locally recurrent probabilistic neural networks with application to text-independent speaker verification , 2007, Neurocomputing.

[38]  D. F. Specht,et al.  Probabilistic neural networks for classification, mapping, or associative memory , 1988, IEEE 1988 International Conference on Neural Networks.

[39]  R. Eberhart,et al.  Comparing inertia weights and constriction factors in particle swarm optimization , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[40]  William S. Meisel,et al.  Computer-oriented approaches to pattern recognition , 1972 .

[41]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[42]  D. F. Specht,et al.  Enhancements to probabilistic neural networks , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[43]  Todor Ganchev,et al.  Locally Recurrent Probabilistic Neural Networks with Application to Speaker Verification , 2004 .

[44]  Jan A Snyman,et al.  Practical Mathematical Optimization: An Introduction to Basic Optimization Theory and Classical and New Gradient-Based Algorithms , 2005 .