Semantic Face Retrieval

The description of a face given by people is almost always semantic in nature using verbal terms such as “long face”, “thick lipped” or “blonde haired”. Existing Face Retrieval systems are mostly image based and hence do not capture these semantic features directly. Further since they perform image matching or other computationally intensive processes for retrieval of faces they are generally inefficient even on relatively small databases of face images. We propose a probabilistic and interactive semantic face retrieval system that retrieves face images based on verbal descriptions given by users. The system is also capable of prompting the user to provide information about facial features of the targeted face that best distinguishes the person from the top choices dynamically at any given time in the query session. The proposed system can supplement systems like Identikit (or other mug-shot retrieval systems) by providing an effective and efficient filter. Since the semantic query process is essentially a table search; the system is efficient and can operate in real-time. The proposed system automates the process of tagging face images with semantic labels which are then used for retrieving images that best fit the verbal description provided by the user. During enrollment, a critical step that needs to be performed for the automated semantic tagging of the face is facial feature localization and parameterization. We present two approaches for the task. First, in cases where mug-shot images are enrolled, primitive vision techniques and heuristics are used to locate and parameterize the facial features. Alternatively, in cases where unconstrained frontal face images are enrolled, a hybrid linear graphical model is proposed to model the relationship between the facial features and their locations based on a training set. This graphical model is then used to locate and parameterize facial feature extraction in a given face image. During the retrieval process Bayesian Inference is used. The system is interactive and prompts the user at each stage of the query process to provide the description of a facial feature that is most discriminative. In our experiments the target face image appeared 77.6% of the time within the top 5 and 90.4% of the time within the top 10 retrieved face images.

[1]  Thomas G. Dietterich What is machine learning? , 2015, Archives of Disease in Childhood.

[2]  G. Pike,et al.  Identifying composites of famous faces: Investigating memory, language and system issues , 2006 .

[3]  Yuchun Fang,et al.  An interactive system for mental face retrieval , 2005, MIR '05.

[4]  Shiqiang Yang,et al.  An HMM-based framework for video semantic analysis , 2005, IEEE Trans. Circuits Syst. Video Technol..

[5]  Venu Govindaraju,et al.  A sampling based approach to facial feature extraction , 2005, Fourth IEEE Workshop on Automatic Identification Advanced Technologies (AutoID'05).

[6]  Horst-Michael Groß,et al.  Classification of Face Images for Gender, Age, Facial Expression, and Identity , 2005, ICANN.

[7]  Sanjeev Khudanpur,et al.  Hidden Markov models for automatic annotation and content-based retrieval of images and video , 2005, SIGIR '05.

[8]  Venu Govindaraju,et al.  A Probabilistic Approach to Semantic Face Retrieval System , 2005, AVBPA.

[9]  James L. Crowley,et al.  Facial features detection robust to pose, illumination and identity , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[10]  Anil K. Jain,et al.  Ethnicity identification from face images , 2004, SPIE Defense + Commercial Sensing.

[11]  H. Ai,et al.  Glasses detection by boosting simple wavelet features , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[12]  Charlie D. Frowd,et al.  EvoFIT: A holistic, evolutionary facial imaging technique for creating composites , 2004, TAP.

[13]  David A. Forsyth,et al.  Automatic Annotation of Everyday Movements , 2003, NIPS.

[14]  Fadi Dornaika,et al.  Efficient active appearance model for real-time head and facial feature tracking , 2003, 2003 IEEE International SOI Conference. Proceedings (Cat. No.03CH37443).

[15]  William T. Freeman,et al.  Nonparametric belief propagation , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[16]  Anil K. Jain,et al.  Semantic face matching , 2002, Proceedings. IEEE International Conference on Multimedia and Expo.

[17]  Javier Ruiz-del-Solar,et al.  FACERET: An Interactive Face Retrieval System Based on Self-Organizing Maps , 2002, CIVR.

[18]  Michael Isard,et al.  Bayesian Object Localisation in Images , 2001, International Journal of Computer Vision.

[19]  Robert Mariani,et al.  Glasses Detection for Face Recognition Using Bayes Rules , 2000, ICMI.

[20]  Shinjiro Kawato,et al.  Real-time detection of nodding and head-shaking by directly detecting and tracking the "between-eyes" , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[21]  Jian-Gang Wang,et al.  Frontal-view face detection and facial feature extraction using color and morphological operations , 1999, Pattern Recognit. Lett..

[22]  R. Dony,et al.  Edge detection on color images using RGB vector angles , 1999, Engineering Solutions for the Next Millennium. 1999 IEEE Canadian Conference on Electrical and Computer Engineering (Cat. No.99TH8411).

[23]  Horst Bunke,et al.  Towards Detection of Glasses in Facial Images , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[24]  Timothy F. Cootes,et al.  Face Recognition Using Active Appearance Models , 1998, ECCV.

[25]  S. Sridharan,et al.  A syntactic approach to automatic lip feature extraction for speaker identification , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[26]  Federico Girosi,et al.  Training support vector machines: an application to face detection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[27]  James L. Crowley,et al.  Multi-modal tracking of faces for video communications , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[28]  Laurenz Wiskott,et al.  Phantom faces for face analysis , 1997, Proceedings of International Conference on Image Processing.

[29]  Mark S. Nixon,et al.  Robust Eye Centre Extraction Using the Hough Transform , 1997, AVBPA.

[30]  Jiri Matas,et al.  Statistical Chromaticity Models for Lip Tracking with B-splines , 1997, AVBPA.

[31]  Alexander H. Waibel,et al.  A real-time face tracker , 1996, Proceedings Third IEEE Workshop on Applications of Computer Vision. WACV'96.

[32]  David C. Gibbon,et al.  Multi-modal system for locating heads and faces , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[33]  Takeo Kanade,et al.  Neural network-based face detection , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[34]  Alex Pentland,et al.  Photobook: Content-based manipulation of image databases , 1996, International Journal of Computer Vision.

[35]  Dragutin Petkovic,et al.  Query by Image and Video Content: The QBIC System , 1995, Computer.

[36]  Larry J. Eshelman,et al.  Proceedings of the 6th International Conference on Genetic Algorithms , 1995 .

[37]  Bülent Sankur,et al.  Facial feature localization and adaptation of a generic face model for model-based coding , 1995, Signal Process. Image Commun..

[38]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[39]  Vijay V. Raghavan,et al.  An approach to interactive retrieval in face image databases based on semantic attributes , 1994 .

[40]  Alex Waibel,et al.  Face locating and tracking for human-computer interaction , 1994, Proceedings of 1994 28th Asilomar Conference on Signals, Systems and Computers.

[41]  Gilles Burel,et al.  Detection and localization of faces on digital images , 1994, Pattern Recognit. Lett..

[42]  Alex Pentland,et al.  View-based and modular eigenspaces for face recognition , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Jian-Kang Wu,et al.  Facial image retrieval, identification, and inference system , 1993, MULTIMEDIA '93.

[44]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[45]  A. Kennedy,et al.  Hybrid Monte Carlo , 1988 .

[46]  Rama Chellappa,et al.  Classification of textures using Gaussian Markov random fields , 1985, IEEE Trans. Acoust. Speech Signal Process..

[47]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  K. Laughery,et al.  Sketch artist and Identi-kit procedures for recalling faces. , 1980 .

[49]  J. Penry Photo-Fit , 1970 .

[50]  T. W. Anderson ASYMPTOTIC THEORY FOR PRINCIPAL COMPONENT ANALYSIS , 1963 .

[51]  Tim Morris,et al.  Computer Vision and Image Processing: 4th International Conference, CVIP 2019, Jaipur, India, September 27–29, 2019, Revised Selected Papers, Part I , 2020, CVIP.

[52]  Geoffrey E. Hinton,et al.  Learning Causally Linked Markov Random Fields , 2005, AISTATS.

[53]  Hermann Ney,et al.  Face-based Image Retrieval – One Step Toward Object-based Image Retrieval ∗ , 2005 .

[54]  Gu Xu,et al.  An HMM-based framework for video semantic analysis , 2005, IEEE Transactions on Circuits and Systems for Video Technology.

[55]  Tamás Szirányi,et al.  Face analysis using CNN-UM , 2004 .

[56]  Fabrice Heitz,et al.  A Bayesian approach to object detection using probabilistic appearance-based models , 2004, Pattern Analysis and Applications.

[57]  In Kyu Park,et al.  AUTOMATIC EXTRACTION OF FRONTAL FACIAL FEATURES , 2003 .

[58]  Mohan S. Kankanhalli,et al.  Semantic Video Annotation and Vague Query , 2003, MMM.

[59]  Michael E. Tipping,et al.  Probabilistic Principal Component Analysis , 1999 .

[60]  Margo Seltzer,et al.  The mug-shot search problem: a study of the eigenface metric, search strategies, and interfaces in a system for searching facial image data , 1999 .

[61]  Aleix M. Martinez,et al.  The AR face database , 1998 .

[62]  A. Martínez,et al.  The AR face databasae , 1998 .

[63]  Shaogang Gong,et al.  Face Tracking and Pose Representation , 1996, BMVC.

[64]  Karin Sobottka Ioannis Pitas Localization Of Facial Regions And Features In Color Images , 1996 .

[65]  Alex Pentland,et al.  Photobook: tools for content-based manipulation of image databases , 1994, Electronic Imaging.

[66]  Radford M. Neal Bayesian training of backpropagation networks by the hybrid Monte-Carlo method , 1992 .

[67]  Stuart German,et al.  Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images , 1988 .

[68]  H. Ellis Recognizing faces. , 1975, British journal of psychology.

[69]  J. Besag Spatial Interaction and the Statistical Analysis of Lattice Systems , 1974 .