Exemplar-based Lip-to-Speech Synthesis Using Convolutional Neural Networks

We propose in this paper a neural network-based lip-to-speech synthesis approach that converts “unvoiced” lip movements to “voiced” utterances. In our previous work, a lip-to-speech conversion method based on exemplar-based nonnegative matrix factorization (NMF) was proposed. However, this approach has several problems. First, the unnatural preprocessing of visual features is required to satisfy the nonnegativity constraint of NMF. Next, there is a possibility that an activity matrix cannot be shared between the visual and the audio feature in an NMF-based approach. To tackle these problems, in this paper, we use convolutional neural networks to convert visual features into audio features. Furthermore, we integrate an exemplar-based approach into the neural networks in order to adopt an advantage associated with our previous work. Experimental results showed that our proposed method produced more natural speech than conventional methods.