Remote Heart Rate Estimation Based on 3D Facial Landmarks

In this paper, we propose a novel video-based remote heart rate (HR) estimation method based on 3D facial landmarks. The key contributions in our method are twofold: (i) We introduce 3D facial landmarks detection to the video-based HR estimation and (ii) we propose a novel face patch visibility check manner based on the face patch normal in the 3D space. We experimentally demonstrate that, compared with baseline methods using 2D facial landmarks, our proposed method using 3D facial landmarks improves the robustness of HR estimation to head rotations and partial face occlusion. We also demonstrate that our visibility check is effective for selecting sufficiently visible face patches, contributing to the improvement of HR estimation accuracy.

[1]  Yu Sun,et al.  Photoplethysmography Revisited: From Contact to Noncontact, From Point to Imaging , 2016, IEEE Transactions on Biomedical Engineering.

[2]  Ashok Veeraraghavan,et al.  DistancePPG: Robust non-contact vital signs monitoring using a camera , 2015, Biomedical optics express.

[3]  Daniel McDuff,et al.  Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam , 2011, IEEE Transactions on Biomedical Engineering.

[4]  H. Emrah Tasli,et al.  Remote PPG based vital sign measurement using adaptive facial regions , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[5]  Stefanos Zafeiriou,et al.  The Menpo Benchmark for Multi-pose 2D and 3D Facial Landmark Localisation and Tracking , 2018, International Journal of Computer Vision.

[6]  Tal Hassner,et al.  On Face Segmentation, Face Swapping, and Face Perception , 2017, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[7]  W. Verkruysse,et al.  Non-contact heart rate monitoring utilizing camera photoplethysmography in the neonatal intensive care unit - a pilot study. , 2013, Early human development.

[8]  J. Tsien,et al.  Remote Measurements of Heart and Respiration Rates for Telemedicine , 2013, PloS one.

[9]  Masatoshi Okutomi,et al.  Remote Heart Rate Measurement from RGB-NIR Video Based on Spatial and Spectral Face Patch Selection , 2018, 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[10]  Horst-Michael Groß,et al.  Non-contact video-based pulse rate measurement on a mobile service robot , 2014, The 23rd IEEE International Symposium on Robot and Human Interactive Communication.

[11]  Georgios Tzimiropoulos,et al.  How Far are We from Solving the 2D & 3D Face Alignment Problem? (and a Dataset of 230,000 3D Facial Landmarks) , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[12]  Sander Stuijk,et al.  Algorithmic Principles of Remote PPG , 2017, IEEE Transactions on Biomedical Engineering.

[13]  Josephine Sullivan,et al.  One millisecond face alignment with an ensemble of regression trees , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Yoshinori Kuno,et al.  Robust Heart Rate Measurement from Video Using Select Random Patches , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[15]  Masatoshi Okutomi,et al.  Inter-Beat Interval Estimation from Facial Video Based on Reliability of BVP Signals , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).