Facial Features Detection and Localization

Detection of facial landmarks and their feature points plays an important role in many facial image-related applications such as face recognition/verification, facial expression analysis, pose normalization, and 3D face reconstruction. Generally, detection of facial features is easy for persons; however, for machines it is not an easy task at all. The difficulty comes from high inter-personal variation (e.g., gender, race), intra-personal changes (e.g., pose, expression), and from acquisition conditions (e.g., lighting, image resolution). This chapter discusses basic concepts related to the problem of facial landmarks detection and overviews the successes and failures of exiting solutions. Also, it explores the difficulties that hinders the path of progress in the topic and the challenges involved in the adaptation of existing approaches to build successful systems that can be utilized in real-world facial images-related applications. Additionally, it discusses the performance evaluation metrics and the available benchmarking datasets. Finally, it suggests some possible future directions for research in the topic.

[1]  Hongwei Ying,et al.  Robust nose tip localization based on two-stage subclass discriminant analysis , 2014, Neurocomputing.

[2]  Luhong Liang,et al.  A detector tree of boosted classifiers for real-time object detection and tracking , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[3]  Hyeonjoon Moon,et al.  The FERET Evaluation Methodology for Face-Recognition Algorithms , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Qiang Ji,et al.  Automatic Eye Detection and Its Validation , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[5]  Maurício Pamplona Segundo,et al.  Automatic Face Segmentation and Facial Landmark Detection in Range Images , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[6]  Fang-Lue Zhang,et al.  A survey of the state-of-the-art in patch-based synthesis , 2017, Computational Visual Media.

[7]  Qiang Ji,et al.  Multi-view face and eye detection using discriminant features , 2007, Comput. Vis. Image Underst..

[8]  Anastasios Delopoulos,et al.  The MUG facial expression database , 2010, 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10.

[9]  Chang-Woo Park,et al.  A robust facial feature detection on mobile robot platform , 2009, Machine Vision and Applications.

[10]  Mahmoud Hassaballah,et al.  A Robust Method for Nose Detection under Various Conditions , 2010, ICCVG.

[11]  Mohan M. Trivedi,et al.  A Binary Tree for Probability Learning in Eye Detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[12]  Andrew Zisserman,et al.  Regression and classification approaches to eye localization in face images , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[13]  Xiaoou Tang,et al.  Learning Deep Representation for Face Alignment with Auxiliary Attributes , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Bir Bhanu,et al.  Facial expression recognition using emotion avatar image , 2011, Face and Gesture 2011.

[15]  Ning Zhang,et al.  A retrieval algorithm for specific face images in airport surveillance multimedia videos on cloud computing platform , 2016, Multimedia Tools and Applications.

[16]  Kin-Man Lam,et al.  An accurate active shape model for facial feature extraction , 2004, Proceedings of 2004 International Symposium on Intelligent Multimedia, Video and Speech Processing, 2004..

[17]  David J. Kriegman,et al.  Localizing Parts of Faces Using a Consensus of Exemplars , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Peter H. N. de With,et al.  Facial feature extraction by a cascade of model-based algorithms , 2008, Signal Process. Image Commun..

[19]  Francisco Javier Díaz Pernas,et al.  Real-time hands, face and facial features detection and tracking: Application to cognitive rehabilitation tests monitoring , 2010, J. Netw. Comput. Appl..

[20]  Wen Gao,et al.  The CAS-PEAL Large-Scale Chinese Face Database and Baseline Evaluations , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[21]  Mohamed Rizon,et al.  Detection of eyes from human faces by Hough transform and separability filter , 2005 .

[22]  Aleix M. Martínez,et al.  Precise Fiducial Detection , 2015, Encyclopedia of Biometrics.

[23]  Mohan M. Trivedi,et al.  Face Expression Recognition by Cross Modal Data Association , 2013, IEEE Transactions on Multimedia.

[24]  Jaihie Kim,et al.  An eye detection method robust to eyeglasses for mobile iris recognition , 2017, Expert Syst. Appl..

[25]  David J. Kriegman,et al.  From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Jing Xiao,et al.  Meticulously detailed eye region model and its application to analysis of facial images , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Tieniu Tan,et al.  Combining local features for robust nose location in 3D facial data , 2006, Pattern Recognit. Lett..

[28]  Yi Yang,et al.  Style Aggregated Network for Facial Landmark Detection , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Maja Pantic,et al.  Facial point detection using boosted regression and graph models , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[30]  Ali Ismail Awad,et al.  Detection and Description of Image Features: An Introduction , 2016 .

[31]  Liya Ding,et al.  Features versus Context: An Approach for Precise and Detailed Detection and Delineation of Faces and Facial Features , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Meng Li,et al.  Automatic lip localization under face illumination with shadow consideration , 2009, Signal Process..

[33]  M. Hassaballah,et al.  Independent components analysis-based nose detection method , 2010, 2010 3rd International Congress on Image and Signal Processing.

[34]  Shiguang Shan,et al.  Robust FEC-CNN: A High Accuracy Facial Landmark Detection System , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[35]  Mahmoud Hassaballah,et al.  Face recognition: challenges, achievements and future directions , 2015, IET Comput. Vis..

[36]  Yuning Jiang,et al.  Extensive Facial Landmark Localization with Coarse-to-Fine Convolutional Network Cascade , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[37]  Heng Yang,et al.  Privileged information-based conditional regression forest for facial feature detection , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[38]  Ioannis Pitas,et al.  Facial feature detection using distance vector fields , 2009, Pattern Recognit..

[39]  Andrea Cavallaro,et al.  Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Stephen J. Cox,et al.  Lip reading from scale-space measurements , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[41]  Reza Safabakhsh,et al.  Human eye sclera detection and tracking using a modified time-adaptive self-organizing map , 2008, Pattern Recognit..

[42]  Bernd Radig,et al.  Adjusted pixel features for robust facial component classification , 2010, Image Vis. Comput..

[43]  Kin-Man Lam,et al.  An efficient algorithm for human face detection and facial feature extraction under different conditions , 2001 .

[44]  Qiang Ji,et al.  Discriminative Deep Face Shape Model for Facial Point Detection , 2015, International Journal of Computer Vision.

[45]  Ioannis Patras,et al.  Sieving Regression Forest Votes for Facial Feature Detection in the Wild , 2013, 2013 IEEE International Conference on Computer Vision.

[46]  Mikkel B. Stegmann,et al.  The IMM Face Database, An Annotated Dataset of 240 Face Images , 2004 .

[47]  Qingshan Liu,et al.  Stacked Hourglass Network for Robust Facial Landmark Localisation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[48]  Dimitris Samaras,et al.  EyeOpener: Editing Eyes in the Wild , 2017, ACM Trans. Graph..

[49]  Jiri Matas,et al.  XM2VTSDB: The Extended M2VTS Database , 1999 .

[50]  Fernando De la Torre,et al.  Supervised Descent Method and Its Applications to Face Alignment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[51]  Zhi-Hua Zhou,et al.  A literature survey on robust and efficient eye localization in real-life scenarios , 2013, Pattern Recognit..

[52]  Kin-Man Lam,et al.  Facial-feature detection and localization based on a hierarchical scheme , 2014, Inf. Sci..

[53]  Claudia Lindner,et al.  Robust and Accurate Shape Model Matching Using Random Forest Regression-Voting. , 2015, IEEE transactions on pattern analysis and machine intelligence.

[54]  Xiaobo Li,et al.  Towards a system for automatic facial feature detection , 1993, Pattern Recognit..

[55]  Witold Pedrycz,et al.  A study in facial regions saliency: a fuzzy measure approach , 2014, Soft Comput..

[56]  Heng Yang,et al.  Facial feature point detection: A comprehensive survey , 2014, Neurocomputing.

[57]  Pong C. Yuen,et al.  Variance projection function and its application to eye detection for human face recognition , 1998, Pattern Recognit. Lett..

[58]  Zheru Chi,et al.  A robust eye detection method using combined binary edge and intensity information , 2006, Pattern Recognit..

[59]  Veikko Surakka,et al.  Automatic edge-based localization of facial features from images with complex facial expressions , 2010, Pattern Recognit. Lett..

[60]  Alan Wee-Chung Liew,et al.  Lip contour extraction from color images using a deformable model , 2002, Pattern Recognit..

[61]  Xiaoou Tang,et al.  Facial Landmark Detection by Deep Multi-task Learning , 2014, ECCV.

[62]  Ian R. Fasel,et al.  A generative framework for real time object detection and classification , 2005, Comput. Vis. Image Underst..

[63]  Timothy F. Cootes,et al.  Overview of research on facial ageing using the FG-NET ageing database , 2016, IET Biom..

[64]  Xi Chen,et al.  Delving Deep Into Coarse-to-Fine Framework for Facial Landmark Localization , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[65]  Vitoantonio Bevilacqua,et al.  Automatic Facial Feature Points Detection , 2008, ICIC.

[66]  Patrick J. Flynn,et al.  Multiple Nose Region Matching for 3D Face Recognition under Varying Facial Expression , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[67]  Xin Fan,et al.  Explicit Shape Regression With Characteristic Number for Facial Landmark Localization , 2018, IEEE Transactions on Multimedia.

[68]  M. E. Celebi,et al.  Advances in Face Detection and Facial Image Analysis , 2016 .

[69]  Bor-Shyh Lin,et al.  Development of Novel Lip-Reading Recognition Algorithm , 2017, IEEE Access.

[70]  Qiong Wang,et al.  Eye Detection in Facial Images With Unconstrained Background , 2006 .

[71]  Pengfei Zhao,et al.  Robust Precise Eye Location by Adaboost and SVM Techniques , 2005, ISNN.

[72]  Veikko Surakka,et al.  Feature-based detection of facial landmarks from neutral and expressive facial images , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[73]  Kenji Murakami,et al.  Face detection evaluation: a new approach based on the golden ratio $${\Phi}$$ , 2013, Signal Image Video Process..

[74]  Su Ruan,et al.  A robust agorithm for eye detection on gray intensity face without spectacles , 2005 .

[75]  Dmitry O. Gorodnichy,et al.  Nouse 'use your nose as a mouse' perceptual vision technology for hands-free games and interfaces , 2004, Image Vis. Comput..

[76]  Harry Wechsler,et al.  Detection of faces and facial landmarks using iconic filter banks , 1997, Pattern Recognit..

[77]  Dan Xu,et al.  Automatic eye detection using intensity filtering and K-means clustering , 2010, Pattern Recognit. Lett..

[78]  Andy Harter,et al.  Parameterisation of a stochastic model for human face identification , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[79]  Hanjiang Lai,et al.  Deep Recurrent Regression for Facial Landmark Detection , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[80]  Paola Campadelli,et al.  Fiducial point localization in color images of face foregrounds , 2004, Image Vis. Comput..

[81]  Pong C. Yuen,et al.  Multi-cues eye detection on gray intensity image , 2001, Pattern Recognit..

[82]  Yasuyuki Nakata,et al.  Lipreading method using color extraction method and eigenspace technique , 2004 .

[83]  Jie Yang,et al.  A robust method for eye features extraction on color image , 2005, Pattern Recognit. Lett..

[84]  Gang Zhang,et al.  Advances and Trends in Video Face Alignment , 2018, Recent Advances in Computer Vision.

[85]  Se-Young Oh,et al.  Automatic extraction of eye and mouth fields from a face image using eigenfeatures and multilayer perceptrons , 2001, Pattern Recognit..

[86]  Alan Wee-Chung Liew,et al.  Segmentation of color lip images by spatial fuzzy clustering , 2003, IEEE Trans. Fuzzy Syst..

[87]  Jie Lin,et al.  Robust face recognition with partial occlusion, illumination variation and limited training data by optimal feature selection , 2011 .

[88]  Nicu Sebe,et al.  What Are You Looking at? , 2011, International Journal of Computer Vision.

[89]  Luc Van Gool,et al.  Hough Transform-based Mouth Localization for Audio-visual Speech Recognition , 2009, BMVC.

[90]  Robert Frischholz,et al.  BioID: A Multimodal Biometric Identification System , 2000, Computer.

[91]  Michael J. Lyons,et al.  Automatic Classification of Single Facial Images , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[92]  Adam Schmidt,et al.  The put face database , 2008 .

[93]  Aleix M. Martínez,et al.  Subclass discriminant analysis , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[94]  Shu Hung Leung,et al.  Lip image segmentation using fuzzy clustering incorporating an elliptic shape function , 2004, IEEE Transactions on Image Processing.

[95]  Michael S. Lew,et al.  Deep learning for visual understanding: A review , 2016, Neurocomputing.

[96]  Klaus J. Kirchberg,et al.  Robust Face Detection Using the Hausdorff Distance , 2001, AVBPA.

[97]  Alan Hanjalic,et al.  Eye localization in low and standard definition content with application to face matching , 2009, Comput. Vis. Image Underst..

[98]  Aleix M. Martinez,et al.  Salient and non-salient fiducial detection using a probabilistic graphical model , 2014, Pattern Recognit..

[99]  Xiaogang Wang,et al.  Deep Convolutional Network Cascade for Facial Point Detection , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[100]  Chengjun Liu,et al.  Eye detection using discriminatory Haar features and a new efficient SVM , 2015, Image Vis. Comput..

[101]  Ke Gu,et al.  Review on Automatic Lip Reading Techniques , 2017, Int. J. Pattern Recognit. Artif. Intell..

[102]  Kamlesh Mistry,et al.  Adaptive facial point detection and emotion recognition for a humanoid robot , 2015, Comput. Vis. Image Underst..

[103]  Wen-Yuan Chen,et al.  Eye detection based on head contour geometry and wavelet subband projection , 2006 .

[104]  Tieniu Tan,et al.  Robust nose detection in 3D facial data using local characteristics , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[105]  Qiang Ji,et al.  A Comparative Study of Local Matching Approach for Face Recognition , 2007, IEEE Transactions on Image Processing.

[106]  Maurício Pamplona Segundo,et al.  3D Face Recognition Using Simulated Annealing and the Surface Interpenetration Measure , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[107]  Alberto Del Bimbo,et al.  Matching 3D face scans using interest points and local histogram descriptors , 2013, Comput. Graph..

[108]  Gang Wang,et al.  Deep Context-Sensitive Facial Landmark Detection With Tree-Structured Modeling , 2017, IEEE Transactions on Image Processing.

[109]  Weihong Deng,et al.  Facial landmark localization by enhanced convolutional neural network , 2018, Neurocomputing.

[110]  Paola Campadelli,et al.  A face recognition system based on automatically determined facial fiducial points , 2006, Pattern Recognit..

[111]  Horst Bischof,et al.  Annotated Facial Landmarks in the Wild: A large-scale, real-world database for facial landmark localization , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[112]  David Cristinacce,et al.  Automatic feature localisation with constrained local models , 2008, Pattern Recognit..

[113]  Kenji Murakami,et al.  Eye and Nose Fields Detection from Gray Scale Facial Images , 2011, MVA.

[114]  Yue Wu,et al.  Learning Pose-Aware Models for Pose-Invariant Face Recognition in the Wild , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[115]  Qiang Ji,et al.  In the Eye of the Beholder: A Survey of Models for Eyes and Gaze , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[116]  Jun Wang,et al.  Eye Detection Under Unconstrained Background by the Terrain Feature , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[117]  Stefanos Zafeiriou,et al.  300 Faces In-The-Wild Challenge: database and results , 2016, Image Vis. Comput..

[118]  Timothy F. Cootes,et al.  Extraction of Visual Features for Lipreading , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[119]  Alan L. Yuille,et al.  Feature extraction from faces using deformable templates , 2004, International Journal of Computer Vision.

[120]  Derek Bradley,et al.  High resolution passive facial performance capture , 2010, SIGGRAPH 2010.

[121]  Zhi-Hua Zhou,et al.  Projection functions for eye detection , 2004, Pattern Recognit..

[122]  Frank Y. Shih,et al.  Automatic extraction of head and face boundaries and facial features , 2004, Inf. Sci..

[123]  M. Hassaballah,et al.  Efficient eye detection method based on grey intensity variance and independent components analysis , 2010 .