SynRhythm: Learning a Deep Heart Rate Estimator from General to Specific

Remote photoplethysmography (rPPG) based noncontact heart rate (HR) measurement from a face video has drawn increasing attention recently because of its potential applications in many scenarios such as training aid, health monitoring, and nursing care. Although a number of methods have been proposed, most of them are designed under certain assumptions and could fail when such assumptions do not hold. At the same time, while deep learning based methods have been reported to achieve promising results in many computer vision tasks, their use in rPPG-based heart rate estimation has been limited due to the very limited data available in public domain. To overcome this limitation and leverage the strong modeling ability of deep neural networks, in this paper, we propose a novel spatial-temporal representation for the HR signal and design a general-to-specific transfer learning strategy to train a deep heart rate estimator from a large volume of synthetic rhythm signals and a limited number of available face video data. Experiment results on the public-domain databases show the effectiveness of the proposed approach.

[1]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[2]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Nicu Sebe,et al.  Self-Adaptive Matrix Completion for Heart Rate Estimation from Face Videos under Realistic Conditions , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[5]  Matti Pietikäinen,et al.  Remote Heart Rate Measurement from Face Videos under Realistic Situations , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Daniel McDuff,et al.  Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam , 2011, IEEE Transactions on Biomedical Engineering.

[7]  Jiri Matas,et al.  All you need is a good init , 2015, ICLR.

[8]  Sander Stuijk,et al.  Exploiting Spatial Redundancy of Image Sensor for Motion Robust rPPG , 2015, IEEE Transactions on Biomedical Engineering.

[9]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[10]  L. O. Svaasand,et al.  Remote plethysmographic imaging using ambient light. , 2008, Optics express.

[11]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[12]  Shiguang Shan,et al.  Continuous heart rate measurement from face: A robust rPPG approach with distribution learning , 2017, 2017 IEEE International Joint Conference on Biometrics (IJCB).

[13]  Anil K. Jain,et al.  Cross-Database Face Antispoofing with Robust Feature Representation , 2016, CCBR.

[14]  Rosalind W. Picard,et al.  Non-contact, automated cardiac pulse measurements using video imaging and blind source separation , 2022 .

[15]  Frédo Durand,et al.  Detecting Pulse from Head Motions in Video , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Yoshinori Kuno,et al.  Robust Heart Rate Measurement from Video Using Select Random Patches , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[17]  Frédo Durand,et al.  Eulerian video magnification for revealing subtle changes in the world , 2012, ACM Trans. Graph..

[18]  Gerard de Haan,et al.  Robust Pulse Rate From Chrominance-Based rPPG , 2013, IEEE Transactions on Biomedical Engineering.

[19]  D. Altman,et al.  STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT , 1986, The Lancet.

[20]  Xin Liu,et al.  AgeNet: Deeply Learned Regressor and Classifier for Robust Apparent Age Estimation , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[21]  Winston H. Hsu,et al.  Learning-based heart rate detection from remote photoplethysmography features , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[22]  Ashok Veeraraghavan,et al.  DistancePPG: Robust non-contact vital signs monitoring using a camera , 2015, Biomedical optics express.

[23]  Gee-Sern Hsu,et al.  Deep learning with time-frequency representation for pulse estimation from facial videos , 2017, 2017 IEEE International Joint Conference on Biometrics (IJCB).

[24]  Kwang Suk Park,et al.  ROI analysis for remote photoplethysmography on facial video , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).