Heterogeneous face recognition

One of the most difficult challenges in automated face recognition is computing facial similarities between face images acquired in alternate modalities. Called heterogeneous face recognition (HFR), successful solutions to this recognition paradigm would allow the vast collection of face photographs (acquired from driver's licenses, passports, mug shots, and other sources of frontal face images) to be matched against face images from alternate modalities (e.g. forensic sketches, infrared images, aged face images). This dissertation offers several contributions to heterogeneous face recognition algorithms. The first contribution is a framework for matching forensic sketches to mug shot photographs. In developing a technique called Local Feature-based Discriminant Analysis (LFDA), we are able to significantly improve sketch recognition accuracies with respect to a state of the art commercial face recognition engine. The improved accuracy of LFDA allows for facial searches of criminal offenders using a hand drawn sketch based on a verbal description of the subject's appearance, called a forensic sketch. The second contribution of this dissertation is a generic framework for heterogeneous face recognition. By representing images from alternate modalities with their non-linear similarity to a set of prototype subjects who provide images from each corresponding modalities, the need to directly compare face images from alternate modality is eliminated. This property generalizes the algorithm, called Prototype Random Subspaces (P-RS), to any HFR scenario. The viability of this algorithm is demonstrated on four separate HFR databases (near infrared, thermal infrared, forensic sketch, and viewed sketch). The third contribution of this dissertation is a large scale examination of face recognition algorithms in the presence of aging. We study whether or not aging-invariant face recognition algorithms generalize to non-aging scenarios. By demonstrating that they do not generalize, we conclude that the heterogeneous appearances between faces that have aged casts aging-invariant face recognition problem in the same category as heterogeneous face recognition. That is, much like images acquired in alternate modalities, aged face images should be matched using specially trained algorithms. The fourth contribution of this dissertation is an examination of how heterogeneous demographics (i.e. gender, race, and age) affect the recognition accuracy of face recognition systems. Using six different face recognition systems (including commercial systems, non-trainable systems, and a trainable face recognition system), the experiments conclude that all systems have a consistently lower recognition accuracy on the following demographic cohorts: (i) females, (ii) black subjects, and (iii) young subjects. This study also examined whether or not recognition accuracy could be improved for a specific demographic cohort by training a system exclusively on that cohort. The fifth contribution of this dissertation is an examination of the problem of identifying a subject from a caricature. A caricature is a facial sketch of a subject's face that exaggerates identifiable facial features beyond realism, yet humans still have a profound ability to identify subjects from their caricature sketch. Automated caricature recognition with the intent of discovering improved facial feature representations with respect to face recognition as a whole. To enable this task, we propose a set of qualitative facial features that encodes the appearance of both caricatures and photographs. We utilized crowdsourcing, to assist in the labeling of the qualitative features. Using these features, we combine logistic regression, multiple kernel learning, and support vector machines to generate a similarity score between a caricature and a facial photograph. Experiments are conducted on a dataset of 196 pairs of caricatures and photographs, which we have made publicly available.