A System Theoretic Approach to Synthesis and Classification of Lip Articulation

We present a system for synthesizing lip movements and recognizing speakers/phrases from visual lip sequences. Low-dimensional geometrical lip features, such as trajectories of landmarks on the outer lip contour and vertical distances for the mouth opening are first extracted from the images. The temporal evolution of these features is modeled with linear dynamical systems, whose parameters are learned using system identification techniques. By carefully exploiting physical constraints of lip movement both in the learning and synthesis stages, realistic synthesis of novel sequences is achieved. Recognition is performed using classification methods, such as nearest neighbors and support vector machines, combined with various metrics based on subspace angles and kernels, such as the Binet-Cauchy, Martin, and Kullback-Leibler kernels. Experiments are designed to find the combination of features, identification method, kernel and classification method that is most appropriate for synthesis and classification of lip articulation.