Remote Pulse Estimation in the Presence of Face Masks

Remote photoplethysmography (rPPG) is a known family of techniques for monitoring blood volume changes from a camera. It may be especially useful for widespread contact-less health monitoring when used to analyze face video from consumer-grade visible-light cameras. The COVID-19 pandemic has caused the widespread use of protective face masks to prevent virus transmission. We found that occlusions from face masks affect face video-based rPPG as the mean absolute error of blood volume estimation is nearly doubled when the face is partially occluded by protective masks. To our knowledge, this paper is the first to analyse the impact of face masks on the accuracy of blood volume pulse estimation and offers several novel elements: (a) two publicly available pulse estimation datasets acquired from 86 unmasked and 61 masked subjects, (b) evaluations of handcrafted algorithms and a 3D convolutional neural network trained on videos of full (unmasked) faces and synthetically generated masks, and (c) data augmentation method (a generator adding a synthetic mask to a face video). Our findings help identify how face masks degrade accuracy of face video analysis, and we discuss paths toward more robust pulse estimation in their presence. The datasets and source codes of all proposed methods are available along with this paper.

[1]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[2]  L. O. Svaasand,et al.  Remote plethysmographic imaging using ambient light. , 2008, Optics express.

[3]  Patrick Grother,et al.  Ongoing Face Recognition Vendor Test (FRVT) Part 6B: Face recognition accuracy with face masks using post-COVID-19 algorithms , 2020 .

[4]  Nicu Sebe,et al.  Self-Adaptive Matrix Completion for Heart Rate Estimation from Face Videos under Realistic Conditions , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Sander Stuijk,et al.  A Novel Algorithm for Remote Photoplethysmography: Spatial Subspace Rotation , 2016, IEEE Transactions on Biomedical Engineering.

[6]  Matti Pietikäinen,et al.  Remote Heart Rate Measurement from Face Videos under Realistic Situations , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Sébastien Marcel,et al.  A Reproducible Study on Remote Heart Rate Measurement , 2017, ArXiv.

[8]  Iasonas Kokkinos,et al.  Describing Textures in the Wild , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  G. Haan,et al.  Improved motion robustness of remote-PPG by using the blood volume pulse signature , 2014, Physiological measurement.

[10]  Daniel McDuff,et al.  DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks , 2018, ECCV.

[11]  Sébastien Marcel,et al.  Pulse-based Features for Face Presentation Attack Detection , 2018, 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS).

[12]  Winston H. Hsu,et al.  Learning-based heart rate detection from remote photoplethysmography features , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[13]  Mika P. Tarvainen,et al.  An advanced detrending method with application to HRV analysis , 2002, IEEE Transactions on Biomedical Engineering.

[14]  Sander Stuijk,et al.  Algorithmic Principles of Remote PPG , 2017, IEEE Transactions on Biomedical Engineering.

[15]  Andrew Zisserman,et al.  Two-Stream Convolutional Networks for Action Recognition in Videos , 2014, NIPS.

[16]  M. Elgendi,et al.  Can Photoplethysmography Replace Arterial Blood Pressure in the Assessment of Blood Pressure? , 2018, Journal of clinical medicine.

[17]  Wenjin Wang,et al.  Analysis of CNN-based remote-PPG to understand limitations and sensitivities , 2020, Biomedical optics express.

[18]  Rosalind W. Picard,et al.  Non-contact, automated cardiac pulse measurements using video imaging and blind source separation , 2022 .

[19]  Lionel Tarassenko,et al.  Non-contact measurement of oxygen saturation with an RGB camera. , 2015, Biomedical optics express.

[20]  Pong C. Yuen,et al.  Remote Photoplethysmography Correspondence Feature for 3D Mask Face Presentation Attack Detection , 2018, ECCV.

[21]  Daniel McDuff,et al.  Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam , 2011, IEEE Transactions on Biomedical Engineering.

[22]  Xilin Chen,et al.  RhythmNet: End-to-End Heart Rate Estimation From Face via Spatial-Temporal Representation , 2019, IEEE Transactions on Image Processing.

[23]  Guoying Zhao,et al.  Remote Photoplethysmograph Signal Measurement from Facial Videos Using Spatio-Temporal Networks , 2019, BMVC.

[24]  Gee-Sern Hsu,et al.  Deep learning with time-frequency representation for pulse estimation from facial videos , 2017, 2017 IEEE International Joint Conference on Biometrics (IJCB).

[25]  Mikhail Kopeliovich,et al.  Remote Photoplethysmography: Rarely Considered Factors , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[26]  Louis-Philippe Morency,et al.  OpenFace 2.0: Facial Behavior Analysis Toolkit , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[27]  Shiguang Shan,et al.  VIPL-HR: A Multi-modal Database for Pulse Estimation from Less-constrained Face Video , 2018, ACCV.

[28]  Gerard de Haan,et al.  Robust Pulse Rate From Chrominance-Based rPPG , 2013, IEEE Transactions on Biomedical Engineering.