End-to-end speech recognition system based on improved CLDNN structure

In the field of end-to-end speech recognition technology based on deep learning, CLDNN (Convolutional Long Short-Term Memory Fully Connected Deep Neural Network) is a commonly used model structure. The fully connected LSTM (Long Short Term Memory) model is used in the traditional CLDNN structure to process the timing information in the speech signal, which is prone to over-fitting during the training process and affects the learning effect. Deeper models tend to perform better, but increasing the model depth by simply stacking the network layers can cause gradient disappearance, gradient explosion, and "degeneration" problems. Aiming at the above phenomena and problems, this paper proposes an improved CLDNN structure. It combines the residual network and ConvLSTM to establish the residual ConvLSTM model, and replaces the fully connected LSTM model in the traditional CLDNN structure. The model structure solves the problems of the traditional CLDNN model, and can increase the model depth by stacking residual ConvLSTM blocks without gradient disappearance, gradient explosion and "degeneration" problems, which makes the speech recognition system perform better. The experimental results show that the model structure has a word error rate (WER) decrease of more than 8% in both Chinese and English speech recognition tasks compared to the traditional CLDNN structure.

[1]  Yu Zhang,et al.  Very deep convolutional networks for end-to-end speech recognition , 2016, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2]  Shinji Watanabe,et al.  Joint CTC-attention based end-to-end speech recognition using multi-task learning , 2016, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[3]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[4]  Dit-Yan Yeung,et al.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting , 2015, NIPS.

[5]  Xiaodong Cui,et al.  Stereo hidden Markov modeling for noise robust speech recognition , 2013, Comput. Speech Lang..

[6]  Tara N. Sainath,et al.  Learning the speech front-end with raw waveform CLDNNs , 2015, INTERSPEECH.

[7]  Francisco Herrera,et al.  Towards Highly Accurate Coral Texture Images Classification Using Deep Convolutional Neural Networks and Data Augmentation , 2018, Expert Syst. Appl..

[8]  Tara N. Sainath,et al.  Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[9]  Erich Elsen,et al.  Deep Speech: Scaling up end-to-end speech recognition , 2014, ArXiv.

[10]  Brian Kingsbury,et al.  Very deep multilingual convolutional neural networks for LVCSR , 2015, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[11]  Navdeep Jaitly,et al.  Towards End-To-End Speech Recognition with Recurrent Neural Networks , 2014, ICML.

[12]  Jian Sun,et al.  Multimodal 2D+3D Facial Expression Recognition With Deep Fusion Convolutional Neural Network , 2017, IEEE Transactions on Multimedia.

[13]  Li Dan,et al.  Speech recognition based on convolutional neural networks , 2016, 2016 IEEE International Conference on Signal and Image Processing (ICSIP).

[14]  Jürgen Schmidhuber,et al.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks , 2006, ICML.

[15]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[16]  Chin-Hui Lee,et al.  Exploiting deep neural networks for detection-based speech recognition , 2013, Neurocomputing.

[17]  Chong Wang,et al.  Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin , 2015, ICML.

[18]  Bhuvana Ramabhadran,et al.  Direct Acoustics-to-Word Models for English Conversational Speech Recognition , 2017, INTERSPEECH.

[19]  Andrew W. Senior,et al.  Long short-term memory recurrent neural network architectures for large scale acoustic modeling , 2014, INTERSPEECH.

[20]  Sylvain Arlot,et al.  A survey of cross-validation procedures for model selection , 2009, 0907.4728.