Learning and recognizing faces: from still images to video sequences

Automatic face recognition is a challenging problem which has received much attention during recent years due to its many applications in different fields such as law enforcement, security applications, human-machine interaction etc. Up to date there is no technique that provides a robust solution for all situations and different applications. From still gray images to face sequences (and passing through color images), this thesis provides new algorithms to learn, detect and recognize faces. It also analyzes some emerging directions such as the integration of facial dynamics in the recognition process. To recognize faces, the thesis proposes a new approach based on Local Binary Patterns (LBP) which consists of dividing the facial image into small regions from which LBP features are extracted and concatenated into a single feature histogram efficiently representing the face image. Then, face recognition is performed using a nearest neighbor classifier in the computed feature space with Chisquare as a dissimilarity metric. The extensive experiments clearly show the superiority of the proposed method over the state-of the-art algorithms on FERET tests. To detect faces, another LBP-based representation which is suitable for low-resolution images, is derived. Using the new representation, a second-degree polynomial kernel SVM classifier is trained to detect frontal faces in complex gray scale images. Experimental results using several complex images show that the proposed approach performs favorably compared to the state-of-art methods. Additionally, experiments with detecting and recognizing low-resolution faces are carried out to demonstrate that the same facial representation can be efficiently used for both the detection and recognition of faces in low-resolution images. To detect faces when the color cue is available, the thesis proposes an approach based on a robust model of skin color, called a skin locus, which is used to extract the skin-like regions. After orientation normalization and based on verifying a set of criteria (face symmetry, presence of some facial features, variance of pixel intensities and connected component arrangement), only facial regions are selected. To learn and visualize faces in video sequences, the recently proposed algorithms for unsupervised learning and dimensionality reduction (LLE and ISOMAP), as well as well known ones (PCA, SOM etc.) are considered and investigated. Some extensions are proposed and a new approach for selecting face models from video sequences is developed. The approach is based on representing the face manifold in a low-dimensional space using the Locally Linear Embedding (LLE) algorithm and then performing K-means clustering. To analyze the emerging direction in face recognition which consists of combining facial shape and dynamic personal characteristics for enhancing face recognition performance, the thesis considers two factors (face sequence length and image quality) and studies their effects on the performance of video-based systems which attempt to use a spatio-temporal representation instead of a still image based one. The extensive experimental results show that motion information enhances automatic recognition but not in a systematic way as in the human visual system. Finally, some key findings of the thesis are considered and used for building a system for access control based on detecting and recognizing faces.