Combined Framework for Real-time Head Pose Estimation using Facial Landmark Detection and Salient Feature Tracking

This paper presents a novel approach to address the head pose estimation (HPE) problem in real world and demanding applications. We propose a new framework that combines the detection of facial landmarks with the tracking of salient features within the head region. That is, rigid facial landmarks are detected from a given face image, while at the same time, salient features are detected within the head region. The 3D coordinates of both set of features result from their intersection on a simple geometric head model (e.g., cylinder or ellipsoid). We then formulate the HPE problem as a perspective-n-point problem that we separately solve by minimizing the reprojection error of each 3D features set and their corresponding facial or salient features in the next face image. The resulting head pose estimations are then combined using Kalman Filter, which allows us to take advantage of the high accuracy when using facial landmarks while enabling us to handle extreme head poses by using salient features. Results are comparable to those from the related literature, with the advantage of being robust under real world situations that might not be covered in the evaluated datasets.

[1]  Jingyu Yang,et al.  A Fast Algorithm Face Detection and Head Pose Estimation for Driver Assistant System , 2006, 2006 8th international Conference on Signal Processing.

[2]  Radu Horaud,et al.  Head pose estimation via probabilistic high-dimensional regression , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[3]  Nicu Sebe,et al.  Combining Head Pose and Eye Location Information for Gaze Estimation , 2012, IEEE Transactions on Image Processing.

[4]  Luc Van Gool,et al.  Random Forests for Real Time 3D Face Analysis , 2012, International Journal of Computer Vision.

[5]  Michael J. Jones,et al.  Real-time 3D head pose and facial landmark estimation from depth images using triangular surface patch features , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Youding Zhu,et al.  Head pose estimation for driver monitoring , 2004, IEEE Intelligent Vehicles Symposium, 2004.

[7]  Josephine Sullivan,et al.  One millisecond face alignment with an ensemble of regression trees , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Mohan M. Trivedi,et al.  Head Pose Estimation in Computer Vision: A Survey , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Tom Drummond,et al.  Faster and Better: A Machine Learning Approach to Corner Detection , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Chao Yin,et al.  Real-time head pose estimation for driver assistance system using low-cost on-board computer , 2016, VRCAI.

[11]  Takeo Kanade,et al.  Pose Robust Face Tracking by Combining Active Appearance Models and Cylinder Head Models , 2007, International Journal of Computer Vision.

[12]  Neil A. Dodgson,et al.  Variation and extrema of human interpupillary distance , 2004, IS&T/SPIE Electronic Imaging.

[13]  Marco La Cascia,et al.  Fast, Reliable Head Tracking under Varying Illumination: An Approach Based on Registration of Texture-Mapped 3D Models , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Takeo Kanade,et al.  Dense 3D face alignment from 2D video for real-time use , 2017, Image Vis. Comput..

[15]  Myung Jin Chung,et al.  3D head tracking and pose-robust 2D texture map-based face recognition using a simple ellipsoid model , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[16]  Takeo Kanade,et al.  Robust 3D Head Tracking by Online Feature Registration , 2008 .

[17]  Simon Lucey,et al.  Deformable Model Fitting by Regularized Landmark Mean-Shift , 2010, International Journal of Computer Vision.

[18]  Ayoub Al-Hamadi,et al.  Boosted human head pose estimation using kinect camera , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[19]  Rita Cucchiara,et al.  POSEidon: Face-from-Depth for Driver Pose Estimation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Vincent Lepetit,et al.  3-D Head Tracking via Invariant Keypoint Learning , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[21]  Simon Baker.  Real-time non-rigid driver head tracking for driver mental state estimation , 2004 .

[22]  Javier R. Movellan,et al.  Generalized adaptive view-based appearance model: Integrated framework for monocular head pose estimation , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[23]  Peter Robinson,et al.  3D Constrained Local Model for rigid and non-rigid facial tracking , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Yoichi Sato,et al.  Pose-Invariant Facial Expression Recognition Using Variable-Intensity Templates , 2007, International Journal of Computer Vision.

[25]  Claire C. Gordon,et al.  2012 Anthropometric Survey of U.S. Army Personnel: Methods and Summary Statistics , 2014 .

[26]  In-So Kweon,et al.  Real-Time Head Orientation from a Monocular Camera Using Deep Neural Network , 2014, ACCV.

[27]  J.-Y. Bouguet,et al.  Pyramidal implementation of the lucas kanade feature tracker , 1999 .

[28]  Roberto Valenti,et al.  Robustifying eye center localization by head pose cues , 2009, CVPR.

[29]  T. Govardhan,et al.  Driver Gaze Tracking And Eyes Off The Road Detection System , 2017 .

[30]  Mohan M. Trivedi,et al.  Head Pose Estimation and Augmented Reality Tracking: An Integrated System and Evaluation for Monitoring Driver Awareness , 2010, IEEE Transactions on Intelligent Transportation Systems.

[31]  Denis Laurendeau,et al.  Highly Accurate and Fully Automatic Head Pose Estimation from a Low Quality Consumer-Level RGB-D Sensor , 2015, HCMC '15.

[32]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[33]  Kostas Karpouzis,et al.  Head pose estimation with one camera, in uncalibrated environments , 2010, EGIHMI '10.

[34]  Luc Van Gool,et al.  Real time head pose estimation with random regression forests , 2011, CVPR 2011.

[35]  B. H. Pawan Prasad,et al.  A robust head pose estimation system for uncalibrated monocular videos , 2010, ICVGIP '10.

[36]  Daijin Kim,et al.  Robust head tracking using 3D ellipsoidal head model in particle filter , 2008, Pattern Recognit..

[37]  Didier Stricker,et al.  Real-time monocular 6-DOF head pose estimation from salient 2D points , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[38]  Jan Kautz,et al.  Robust Model-Based 3D Head Pose Estimation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[39]  Rainer Stiefelhagen,et al.  DriveAHead — A Large-Scale Driver Head Pose Dataset , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[40]  Wei Liang,et al.  3D head pose estimation with convolutional neural network trained on synthetic images , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[41]  Mohan M. Trivedi,et al.  Multi-spectral and multi-perspective video arrays for driver body tracking and activity analysis , 2007, Comput. Vis. Image Underst..

[42]  T. Kanade,et al.  Robust 3 D Head Tracking by View-based Feature Point Registration , 2010 .

[43]  Robert A. Walker,et al.  Anthropometric Survey of U.S. Army Personnel: Summary Statistics, Interim Report for 1988 , 1989 .

[44]  Bok-Suk Shin,et al.  Novel Backprojection Method for Monocular Head Pose Estimation , 2013, Int. J. Fuzzy Log. Intell. Syst..