Towards stabilizing facial landmark detection and tracking via hierarchical filtering: A new method

Abstract Many facial landmark detection and tracking methods suffer from instability problems that have a negative influence on real-world applications, such as facial animation, head pose estimation and real-time facial 3D reconstruction. The instability results of landmark tracking cause face pose shaking and face movement that is not fluent enough. However, most of the existing landmark detection and tracking methods only consider the stability of face location but neglect the stability of local landmark movement. To solve the problem of landmark local shaking, we present a novel hierarchical filtering method for stabilized facial landmark detection and tracking in video frames. The proposed method addresses the challenging landmark local shaking problem and provides effective remedies to solve them. The main contribution within our solution is a novel hierarchical filtering strategy, which guarantees the robustness of global whole facial shape tracking and the adaptivity of local facial parts tracking. The proposed solution does not depend on specific face detection and alignment algorithms, and thus, can be easily deployed into existing systems. Extensive experimental evaluations and analyses on benchmark datasets and 3D head pose datasets verify the effectiveness of our proposed stabilizing method.

[1]  Stefanos Zafeiriou,et al.  From Pixels to Response Maps: Discriminative Image Filtering for Face Alignment in the Wild , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Václav Hlavác,et al.  Facial Landmark Tracking by Tree-Based Deformable Part Model Based Detector , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[3]  Stefanos Zafeiriou,et al.  Incremental Face Alignment in the Wild , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Timothy F. Cootes,et al.  Feature Detection and Tracking with Constrained Local Models , 2006, BMVC.

[5]  Davis E. King.  Max-Margin Object Detection , 2015, ArXiv.

[6]  George Trigeorgis,et al.  The 3D Menpo Facial Landmark Tracking Challenge , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[7]  Junzhou Huang,et al.  Face Landmark Fitting via Optimized Part Mixtures and Cascaded Deformable Model , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Kun Zhou,et al.  3D shape regression for real-time facial animation , 2013, ACM Trans. Graph..

[9]  Qingshan Liu,et al.  Facial Shape Tracking via Spatio-Temporal Cascade Shape Regression , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[10]  Simon Lucey,et al.  Deformable Model Fitting by Regularized Landmark Mean-Shift , 2010, International Journal of Computer Vision.

[11]  William J. Christmas,et al.  A Multiresolution 3D Morphable Face Model and Fitting Framework , 2016, VISIGRAPP.

[12]  Jian Sun,et al.  Face Alignment at 3000 FPS via Regressing Local Binary Features , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Yiying Tong,et al.  Adaptive 3D Face Reconstruction from Unconstrained Photo Collections , 2016, CVPR.

[14]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[15]  Guoqing Li,et al.  Combining Local and Global Features for 3D Face Tracking , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[16]  Marios Savvides,et al.  Automatic Facial Landmark Tracking in Video Sequences Using Kalman Filter Assisted Active Shape Models , 2010, ECCV Workshops.

[17]  Sébastien Marcel,et al.  On the effectiveness of local binary patterns in face anti-spoofing , 2012, 2012 BIOSIG - Proceedings of the International Conference of Biometrics Special Interest Group (BIOSIG).

[18]  Stefanos Zafeiriou,et al.  The First Facial Landmark Tracking in-the-Wild Challenge: Benchmark and Results , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[19]  Marco La Cascia,et al.  Fast, Reliable Head Tracking under Varying Illumination: An Approach Based on Registration of Texture-Mapped 3D Models , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Sami Romdhani,et al.  Face identification across different poses and illuminations with a 3D morphable model , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[21]  Xiangyu Zhu,et al.  Face Alignment in Full Pose Range: A 3D Total Solution , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Timothy F. Cootes,et al.  Active Appearance Models , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Peter Robinson,et al.  Constrained Local Neural Fields for Robust Facial Landmark Detection in the Wild , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[24]  Kun Zhou,et al.  Displaced dynamic expression regression for real-time facial tracking and animation , 2014, ACM Trans. Graph..

[25]  Xiaoming Liu,et al.  Large-Pose Face Alignment via CNN-Based Dense 3D Model Fitting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Fernando De la Torre,et al.  Supervised Descent Method and Its Applications to Face Alignment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Josephine Sullivan,et al.  One millisecond face alignment with an ensemble of regression trees , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Stefanos Zafeiriou,et al.  A survey on face detection in the wild: Past, present and future , 2015, Comput. Vis. Image Underst..

[29]  Peter Robinson,et al.  OpenFace: An open source facial behavior analysis toolkit , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[30]  Mark Pauly,et al.  Phace: physics-based face modeling and animation , 2017, ACM Trans. Graph..

[31]  Michel F. Valstar,et al.  Cascaded Continuous Regression for Real-time Incremental Face Tracking , 2016, ECCV.

[32]  Thomas Vetter,et al.  Face Recognition Based on Fitting a 3D Morphable Model , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  Xiaoou Tang,et al.  Learning Deep Representation for Face Alignment with Auxiliary Attributes , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.