Problems associated with current area-based visual speech feature extraction techniques

Techniques such as principle component analysis (PCA), linear discriminant analysis (LDA) and the discrete cosine transform (DCT) have all been used to good effect in face recognition. As these techniques are able to compactly represent a set of features, researchers have sought to use these methods to extract the visual speech content for audio-visual speech recognition (AVSR). In this paper, we expose the problems of employing such techniques in AVSR by running some visual-only speech recognition experiments. The results of these experiments illustrate that current area-based feature extraction techniques are heavily dependent on the visual front-end, as well as being ineffective in decoupling adequate speech content from a speaker’s mouth. As a potential solution, we introduce the concept of a free-parts representation, which may be able to circumvent many of these problems experienced by current area-based techniques.