A minimum converted trajectory error (MCTE) approach to high quality speech-to-lips conversion

High quality speech-to-lips conversion, investigated in t his work, renders realistic lips movement (video) consistent with input sp eech (audio) without knowing its linguistic content. Instead of memoryless framebased conversion, we adopt maximum likelihood estimation of the visual parameter trajectories using an audio-visual joint Gau ssian Mixture Model (GMM). We propose a minimum converted trajectory error approach (MCTE) to further refine the converted visual paramete rs. First, we reduce the conversion error by training the joint audio-visual GMM with weighted audio and visual likelihood. Then MCTE uses the generalized probabilistic descent algorithm to minimize a conversion error of the visual parameter trajectories defined on the optimal Gau ssian kernel sequence according to the input speech. We demonstrate the effectiveness of the proposed methods using the LIPS 2009 Visual Speech Synthesis Challenge dataset, without knowing the linguistic (phonetic) content of the input speech. Index Terms: visual speech synthesis, speech-to-lips conversion, mini mum conversion error, minimum generation error