Hierarchical On-line Appearance-Based Tracking for 3D head pose, eyebrows, lips, eyelids and irises

In this paper, we propose an On-line Appearance-Based Tracker (OABT) for simultaneous tracking of 3D head pose, lips, eyebrows, eyelids and irises in monocular video sequences. In contrast to previously proposed tracking approaches, which deal with face and gaze tracking separately, our OABT can also be used for eyelid and iris tracking, as well as 3D head pose, lips and eyebrows facial actions tracking. Furthermore, our approach applies an on-line learning of changes in the appearance of the tracked target. Hence, the prior training of appearance models, which usually requires a large amount of labeled facial images, is avoided. Moreover, the proposed method is built upon a hierarchical combination of three OABTs, which are optimized using a Levenberg-Marquardt Algorithm (LMA) enhanced with line-search procedures. This, in turn, makes the proposed method robust to changes in lighting conditions, occlusions and translucent textures, as evidenced by our experiments. Finally, the proposed method achieves head and facial actions tracking in real-time.

[1]  Hongbin Zha,et al.  Eye state detection from color facial image sequence , 2002, Other Conferences.

[2]  J. Mixter.  Fast , 2012 .

[3]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[4]  Jing Xiao,et al.  Meticulously detailed eye model and its application to analysis of facial image , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[5]  Jing Xiao,et al.  Meticulously detailed eye region model and its application to analysis of facial images , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  J. Cohn,et al.  Automatic recognition of eye blinking in spontaneously occurring behavior , 2002 .

[7]  Timothy F. Cootes,et al.  Statistical models of appearance for computer vision , 1999 .

[8]  Simon Baker,et al.  Active Appearance Models Revisited , 2004, International Journal of Computer Vision.

[9]  Nicu Sebe,et al.  Improving Visual Gaze Estimation by Saliency , 2012 .

[10]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[11]  Ralph Gross,et al.  Generic vs. person specific active appearance models , 2005, Image Vis. Comput..

[12]  Hongbin Zha,et al.  A new method of detecting human eyelids based on deformable templates , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[13]  Frederick R. Forst,et al.  On robust estimation of the location parameter , 1980 .

[14]  Azriel Rosenfeld,et al.  A method of detecting and tracking irises and eyelids in video , 2002, Pattern Recognit..

[15]  Jörgen Ahlberg,et al.  CANDIDE-3 - An Updated Parameterised Face , 2001 .

[16]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  David Fernández Llorca,et al.  Face tracking with automatic model construction , 2011, Image Vis. Comput..

[18]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[19]  Richard Bowden,et al.  Robust Facial Feature Tracking Using Shape-Constrained Multiresolution-Selected Linear Predictors , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Fred Nicolls,et al.  Locating Facial Features with an Extended Active Shape Model , 2008, ECCV.

[21]  Jörgen Ahlberg,et al.  An Active Model for Facial Feature Tracking , 2002, EURASIP J. Adv. Signal Process..

[22]  Marco La Cascia,et al.  Fast, Reliable Head Tracking under Varying Illumination: An Approach Based on Registration of Texture-Mapped 3D Models , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  David Clark.  Comparing Huber's M-Estimator Function with the Mean Square Error in Backpropagation Networks when the Training Data is Noisy , 2000 .

[24]  B. Heisele.  Face Detection , 2001 .

[25]  Nicu Sebe,et al.  What Are You Looking at? , 2011, International Journal of Computer Vision.

[26]  Iain Matthews,et al.  Passive Driver Gaze Tracking with Active Appearance Models (特集 センシング技術) , 2004 .

[27]  Jing Xiao,et al.  Automatic recognition of eye blinking in spontaneously occurring behavior , 2002, Object recognition supported by user interaction for service robots.

[28]  David J. Fleet,et al.  Robust Online Appearance Models for Visual Tracking , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Yiannis Aloimonos,et al.  Perspective approximations , 1990, Image Vis. Comput..

[30]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[31]  Maja Pantic,et al.  Facial point detection using boosted regression and graph models , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[32]  Xiaoming Liu,et al.  Video-based face model fitting using Adaptive Active Appearance Model , 2010, Image Vis. Comput..

[33]  Deva Ramanan,et al.  Face detection, pose estimation, and landmark localization in the wild , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Guillaume S. Masson,et al.  Motion perception during saccadic eye movements , 2000, Nature Neuroscience.

[35]  F. Xavier Roca,et al.  Real-time gaze tracking with appearance-based models , 2009, Machine Vision and Applications.

[36]  Maja Pantic,et al.  Local Evidence Aggregation for Regression-Based Facial Point Detection , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[38]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[39]  Daijin Kim,et al.  Adaptive active appearance model with incremental learning , 2009, Pattern Recognit. Lett..

[40]  J. Shewchuk.  An Introduction to the Conjugate Gradient Method Without the Agonizing Pain , 1994 .

[41]  D. E. Redmond,et al.  Spontaneous Blink Rates Correlate with Dopamine Levels in the Caudate Nucleus of MPTP-Treated Monkeys , 1999, Experimental Neurology.

[42]  Radek Grzeszczuk,et al.  A data-driven model for monocular face tracking , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[43]  Fadi Dornaika,et al.  Real time 3D face and facial feature tracking , 2007, Journal of Real-Time Image Processing.

[44]  Takahiro Ishikawa,et al.  The template update problem , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.