A literature survey on robust and efficient eye localization in real-life scenarios

Eye localization has gained a wide range of applications in face recognition, gaze estimation, pose estimation, expression analysis, etc. However, due to the high degree of appearance variability of eyes in size, shape, color, texture and various ambient environment changes, this task is challenging. During the last three decades, numerous techniques have been developed to meet these challenges. The goal of this paper is to categorize and evaluate these algorithms in a comprehensive way. We focus on the overall difficulties and challenges in real-life scenarios, and present a detailed review of prominent algorithms from the perspective of learning generalizable, flexible and efficient statistical eye models from a small number of training images. In addition, we organize the discussion of the global aspects of eye localization in uncontrolled environments, towards the development of a robust eye localization system. This paper concludes with several promising directions for future research.

[1]  Fernando De la Torre,et al.  Facial feature detection with optimal pixel reduction SVM , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[2]  Jun Wang,et al.  Using geometric properties of topographic manifold to detect and track eyes for human-computer interaction , 2007, TOMCCAP.

[3]  Alan L. Yuille,et al.  Feature extraction from faces using deformable templates , 2004, International Journal of Computer Vision.

[4]  D. Casasent,et al.  Minimum average correlation energy filters. , 1987, Applied optics.

[5]  Zhi-Hua Zhou,et al.  Projection functions for eye detection , 2004, Pattern Recognit..

[6]  Erik G. Learned-Miller,et al.  Online domain adaptation of a pre-trained cascade of classifiers , 2011, CVPR 2011.

[7]  Ioannis Pitas,et al.  Facial feature detection using distance vector fields , 2009, Pattern Recognit..

[8]  Ravi Kothari,et al.  Detection of eye locations in unconstrained visual images , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[9]  Ville Ojansivu,et al.  Blur Insensitive Texture Classification Using Local Phase Quantization , 2008, ICISP.

[10]  Shengcai Liao,et al.  Illumination Invariant Face Recognition Using Near-Infrared Images , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Christoph H. Lampert,et al.  Learning to Localize Objects with Structured Output Regression , 2008, ECCV.

[12]  Wen Gao,et al.  Locally Assembled Binary (LAB) feature with feature-centric cascade for fast and accurate face detection , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Mikkel B. Stegmann,et al.  The IMM Face Database, An Annotated Dataset of 240 Face Images , 2004 .

[14]  Bin Li,et al.  Precise Eye Localization with AdaBoost and Fast Radial Symmetry , 2006, CIS.

[15]  Theo Gevers,et al.  Accurate eye center location and tracking using isophote curvature , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Pong C. Yuen,et al.  Variance projection function and its application to eye detection for human face recognition , 1998, Pattern Recognit. Lett..

[17]  Norbert Krüger,et al.  Face Recognition by Elastic Bunch Graph Matching , 1997, CAIP.

[18]  Joe Marques,et al.  Effects of Eye Position on Eigenface-Based Face Recognition Scoring , 2003 .

[19]  Stan Z. Li,et al.  A robust eye localization method for low quality face images , 2011, 2011 International Joint Conference on Biometrics (IJCB).

[20]  Hyeonjoon Moon,et al.  The FERET Evaluation Methodology for Face-Recognition Algorithms , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  S. Ullman.  Visual routines , 1984, Cognition.

[22]  Qiang Ji,et al.  Automatic Eye Detection and Its Validation , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[23]  Shree K. Nayar,et al.  Attribute and simile classifiers for face verification , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[24]  Pengfei Zhao,et al.  Robust Precise Eye Location by Adaboost and SVM Techniques , 2005, ISNN.

[25]  Liming Zhang,et al.  Spatio-temporal Saliency detection using phase spectrum of quaternion fourier transform , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  T. Boult,et al.  The eyes have it , 2003, WBMA '03.

[27]  Jian-Gang Wang,et al.  Frontal-view face detection and facial feature extraction using color and morphological operations , 1999, Pattern Recognit. Lett..

[28]  Wen Gao,et al.  Curse of mis-alignment in face recognition: problem and a novel mis-alignment learning solution , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[29]  Jiri Matas,et al.  Feature-based affine-invariant localization of faces , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Thomas Hofmann,et al.  Predicting Structured Data (Neural Information Processing) , 2007 .

[31]  Paola Campadelli,et al.  Eye localization: a survey , 2007 .

[32]  Wen Gao,et al.  A novel pupil localization method based on GaborEye model and radial symmetry operator , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[33]  Andrew Zisserman,et al.  Regression and classification approaches to eye localization in face images , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[34]  Yi Zhang,et al.  A Pupil Localization Algorithm Based on Adaptive Gabor Filtering and Negative Radial Symmetry , 2007, EMMCVPR.

[35]  Jian Sun,et al.  Face recognition with learning-based descriptor , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[36]  Paola Campadelli,et al.  Fiducial point localization in color images of face foregrounds , 2004, Image Vis. Comput..

[37]  Xin Jin,et al.  Face alignment using local hough voting , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[38]  Javier Ruiz-del-Solar,et al.  Recognition of Faces in Unconstrained Environments: A Comparative Study , 2009, EURASIP J. Adv. Signal Process..

[39]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[40]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Takeo Kanade,et al.  Neural Network-Based Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Deva Ramanan,et al.  Face detection, pose estimation, and landmark localization in the wild , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[44]  Zia-ur Rahman,et al.  A multiscale retinex for bridging the gap between color images and the human observation of scenes , 1997, IEEE Trans. Image Process..

[45]  Anil K. Jain,et al.  Face Detection in Color Images , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  David J. Kriegman,et al.  What Is the Set of Images of an Object Under All Possible Illumination Conditions? , 1998, International Journal of Computer Vision.

[47]  Alberto Del Bimbo,et al.  Real-time head tracking from the deformation of eye contours using a piecewise affine camera , 1999, Pattern Recognit. Lett..

[48]  Ning Wang,et al.  Robust precise eye location under probabilistic framework , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[49]  Terrance E. Boult,et al.  For your eyes only , 2012, 2012 IEEE Workshop on the Applications of Computer Vision (WACV).

[50]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[51]  Patrick J. Flynn,et al.  Overview of the face recognition grand challenge , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[52]  Pushmeet Kohli,et al.  On Detection of Multiple Object Instances Using Hough Transforms , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[53]  Fei Yang,et al.  Eye localization through multiscale sparse dictionaries , 2011, Face and Gesture 2011.

[54]  Xiaobo Li,et al.  Towards a system for automatic facial feature detection , 1993, Pattern Recognit..

[55]  Cordelia Schmid,et al.  A Performance Evaluation of Local Descriptors , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[56]  He Yan,et al.  Robust Facial Features Localization on Rotation Arbitrary Multi-View face in Complex Background , 2011, J. Comput..

[57]  Wenhui Li,et al.  Eye Location via a Novel Integral Projection Function and Radial Symmetry Transform , 2011 .

[58]  Takeo Kanade,et al.  Dual-state parametric eye tracking , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[59]  Sanghoon Kim,et al.  Multi-Scale Gabor Feature Based Eye Localization , 2007 .

[60]  Zhi-Hua Zhou,et al.  Enhanced Pictorial Structures for precise eye localization under incontrolled conditions , 2009, CVPR.

[61]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[62]  Bernt Schiele,et al.  Robust Object Detection with Interleaved Categorization and Segmentation , 2008, International Journal of Computer Vision.

[63]  Wen Gao,et al.  2D Cascaded AdaBoost for Eye Localization , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[64]  Subhransu Maji,et al.  Max-margin additive classifiers for detection , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[65]  Paola Campadelli,et al.  Precise Eye and Mouth Localization , 2009, Int. J. Pattern Recognit. Artif. Intell..

[66]  Osamu Yamaguchi,et al.  Facial feature localization using weighted vector concentration approach , 2010, Image Vis. Comput..

[67]  Ran Xu,et al.  Human detection in images via L1-norm Minimization Learning , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[68]  Klaus J. Kirchberg,et al.  Robust Face Detection Using the Hausdorff Distance , 2001, AVBPA.

[69]  Shinjiro Kawato,et al.  Real-time detection of nodding and head-shaking by directly detecting and tracking the "between-eyes" , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[70]  Bruce A. Draper,et al.  Average of Synthetic Exact Filters , 2009, CVPR.

[71]  Daniel P. Huttenlocher,et al.  Pictorial Structures for Object Recognition , 2004, International Journal of Computer Vision.

[72]  Christophe Garcia,et al.  Convolutional face finder: a neural architecture for fast and robust face detection , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[73]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[74]  B. Heisele.  Face Detection , 2001 .

[75]  Alan Hanjalic,et al.  Eye localization in low and standard definition content with application to face matching , 2009, Comput. Vis. Image Underst..

[76]  Vennila Ramalingam,et al.  Facial expression recognition - A real time approach , 2009, Expert Syst. Appl..

[77]  Zhiwei Zhu,et al.  Real-time eye detection and tracking under various light conditions , 2002, ETRA.

[78]  Peyman Milanfar,et al.  Training-Free, Generic Object Detection Using Locally Adaptive Regression Kernels , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[79]  B. V. Vijaya Kumar,et al.  Unconstrained correlation filters. , 1994, Applied optics.

[80]  Zhi-Hua Zhou,et al.  Enhanced Pictorial Structures for precise eye localization under incontrolled conditions , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[81]  Shree K. Nayar,et al.  FaceTracer: A Search Engine for Large Collections of Images with Faces , 2008, ECCV.

[82]  Beat Fasel,et al.  Automati Fa ial Expression Analysis: A Survey , 1999 .

[83]  Alex Pentland,et al.  View-based and modular eigenspaces for face recognition , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[84]  Paola Campadelli,et al.  Precise Eye Localization through a General-to-specific Model Definition , 2006, BMVC.

[85]  Qiang Ji,et al.  In the Eye of the Beholder: A Survey of Models for Eyes and Gaze , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[86]  Luc Van Gool,et al.  Fast PRISM: Branch and Bound Hough Transform for Object Class Detection , 2011, International Journal of Computer Vision.

[87]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[88]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[89]  Azriel Rosenfeld,et al.  Face recognition: A literature survey , 2003, CSUR.

[90]  Matti Pietikäinen,et al.  A comparative study of texture measures with classification based on featured distributions , 1996, Pattern Recognit..

[91]  Horst Bischof,et al.  Annotated Facial Landmarks in the Wild: A large-scale, real-world database for facial landmark localization , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[92]  Harry Wechsler,et al.  Visual routines for eye location using learning and evolution , 2000, IEEE Trans. Evol. Comput..

[93]  Kazuo Kyuma,et al.  Face Recognition System Using Local Autocorrelations and Multiscale Integration , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[94]  Václav Hlavác,et al.  Detector of Facial Landmarks Learned by the Structured Output SVM , 2012, VISAPP.

[95]  Zhouyu Fu,et al.  On Mixtures of Linear SVMs for Nonlinear Classification , 2008, SSPR/SPR.

[96]  Feipei Lai,et al.  Region-based template deformation and masking for eye-feature extraction and description , 1997, Pattern Recognit..

[97]  Hong Yan,et al.  Locating and extracting the eye in human face images , 1996, Pattern Recognit..

[98]  Xiaoyang Tan,et al.  Enhanced Local Texture Feature Sets for Face Recognition Under Difficult Lighting Conditions , 2007, IEEE Transactions on Image Processing.

[99]  Christoph H. Lampert,et al.  Beyond sliding windows: Object localization by efficient subwindow search , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[100]  Subhransu Maji,et al.  Classification using intersection kernel support vector machines is efficient , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[101]  Hanqi Zhuang,et al.  On improving eye feature extraction using deformable templates , 1994, Pattern Recognit..

[102]  Alex Pentland,et al.  Probabilistic Visual Learning for Object Representation , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[103]  Dan Xu,et al.  Automatic eye detection using intensity filtering and K-means clustering , 2010, Pattern Recognit. Lett..

[104]  Shimon Ullman,et al.  Face Recognition: The Problem of Compensating for Changes in Illumination Direction , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[105]  Ralph Gross,et al.  An Image Preprocessing Algorithm for Illumination Invariant Face Recognition , 2003, AVBPA.

[106]  Matti Pietikäinen,et al.  Face Description with Local Binary Patterns: Application to Face Recognition , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[107]  Min-Jung Kim,et al.  Jaceosidin Induces Apoptosis in ras‐Transformed Human Breast Epithelial Cells through Generation of Reactive Oxygen Species , 2007, Annals of the New York Academy of Sciences.

[108]  Simon Lucey,et al.  Deformable model fitting with a mixture of local experts , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[109]  Bernt Schiele,et al.  An Implicit Shape Model for Combined Object Categorization and Segmentation , 2006, Toward Category-Level Object Recognition.

[110]  Mark S. Nixon,et al.  Eye Spacing Measurement for Facial Recognition , 1985, Optics & Photonics.

[111]  James L. Crowley,et al.  Facial features detection robust to pose, illumination and identity , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[112]  Martin A. Fischler,et al.  The Representation and Matching of Pictorial Structures , 1973, IEEE Transactions on Computers.

[113]  Roberto Brunelli,et al.  Face Recognition: Features Versus Templates , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[114]  David J. Kriegman,et al.  Acquiring linear subspaces for face recognition under variable lighting , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[115]  Erik Hjelmås,et al.  Face Detection: A Survey , 2001, Comput. Vis. Image Underst..

[116]  Loris Nanni,et al.  Combining Face and Eye Detectors in a High- Performance Face-Detection System , 2012, IEEE MultiMedia.

[117]  Erik G. Learned-Miller,et al.  Unsupervised Joint Alignment of Complex Images , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[118]  Zhiwei Zhu,et al.  Robust real-time eye detection and tracking under variable lighting conditions and various face orientations , 2005, Comput. Vis. Image Underst..