Visual Heart Rate Estimation with Convolutional Neural Network

We propose a novel two-step convolutional neural network to estimate a heart rate from a sequence of facial images. The network is trained end-to-end by alternating optimization and validated on three publicly available datasets yielding state-of-the-art results against three baseline methods. The network performs better by a 40% margin to the state-of-the-art method on a newly collected dataset. A challenging dataset of 204 fitness-themed videos is introduced. The dataset is designed to test the robustness of heart rate estimation methods to illumination changes and subject’s motion. 17 subjects perform 4 activities (talking, rowing, exercising on a stationary bike and an elliptical trainer) in 3 lighting setups. Each activity is captured by two RGB web-cameras, one is placed on a tripod, the other is attached to the fitness machine which vibrates significantly. Subject’s age ranges from 20 to 53 years, the mean heart rate is ≈ 110, the standard deviation ≈ 25.

[1]  L. O. Svaasand,et al.  Remote plethysmographic imaging using ambient light. , 2008, Optics express.

[2]  Gerard de Haan,et al.  Robust Pulse Rate From Chrominance-Based rPPG , 2013, IEEE Transactions on Biomedical Engineering.

[3]  Nicu Sebe,et al.  Self-Adaptive Matrix Completion for Heart Rate Estimation from Face Videos under Realistic Conditions , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[5]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[6]  Matti Pietikäinen,et al.  Remote Heart Rate Measurement from Face Videos under Realistic Situations , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Sébastien Marcel,et al.  A Reproducible Study on Remote Heart Rate Measurement , 2017, ArXiv.

[8]  Sepp Hochreiter,et al.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , 2015, ICLR.

[9]  David Fofi,et al.  Heart rate estimation using facial video: A review , 2017, Biomed. Signal Process. Control..

[10]  Horst-Michael Groß,et al.  Non-contact video-based pulse rate measurement on a mobile service robot , 2014, The 23rd IEEE International Symposium on Robot and Human Interactive Communication.

[11]  S. Barker,et al.  Effects of motion, ambient light, and hypoperfusion on pulse oximeter function , 1997 .

[12]  Yadong Wang,et al.  A review of non-contact, low-cost physiological information measurement based on photoplethysmographic imaging , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[13]  Yu Sun,et al.  Photoplethysmography Revisited: From Contact to Noncontact, From Point to Imaging , 2016, IEEE Transactions on Biomedical Engineering.

[14]  G Mardirossian,et al.  Limitations of pulse oximetry. , 1992, Anesthesia progress.

[15]  Sander Stuijk,et al.  Exploiting Spatial Redundancy of Image Sensor for Motion Robust rPPG , 2015, IEEE Transactions on Biomedical Engineering.

[16]  Liang-Gee Chen,et al.  Robust heart rate measurement with phonocardiogram by on-line template extraction and matching , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[17]  Daniel McDuff,et al.  The Impact of Video Compression on Remote Cardiac Pulse Measurement Using Imaging Photoplethysmography , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[18]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[19]  Sander Stuijk,et al.  A Novel Algorithm for Remote Photoplethysmography: Spatial Subspace Rotation , 2016, IEEE Transactions on Biomedical Engineering.

[20]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[21]  X. Teng,et al.  The effect of contacting force on photoplethysmographic signals , 2004, Physiological measurement.

[22]  Jiri Matas,et al.  WaldBoost - learning for time constrained sequential detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).