Phone-aware LSTM-RNN for voice conversion

This paper investigates a new voice conversion technique using phone-aware Long Short-Term Memory Recurrent Neural Networks (LSTM-RNNs). Most existing voice conversion methods, including Joint Density Gaussian Mixture Models (JDGMMs), Deep Neural Networks (DNNs) and Bidirectional Long Short-Term Memory Recurrent Neural Networks (BLSTM-RNNs), only take acoustic information of speech as features to train models. We propose to incorporate linguistic information to build voice conversion system by using monophones generated by a speech recognizer as linguistic features. The monophones and spectral features are combined together to train LSTM-RNN based voice conversion models, reinforcing the context-dependency modelling of LSTM-RNNs. The results of the 1st voice conversion challenge shows our system achieves significantly higher performance than baseline (GMM method) and was found among the most competitive scores in similarity test. Meanwhile, the experimental results show phone-aware LSTM-RNN method obtains lower Mel-cepstral distortion and higher MOS scores than the baseline LSTM-RNNs.