Exploring the feasibility of seamless remote heart rate measurement using multiple synchronized cameras

Heart rate (HR) measurement and monitoring is of great importance to determine the physiological and mental status of individuals. Recently, it has been demonstrated that HR can be remotely retrieved from facial video-based photoplethysmographic signals captured using consumer-grade cameras. However, in existing studies, subjects are mostly required to keep their facial regions of interest (ROIs) within one single camera. To make this technique usable in a daily life situation where subjects move around freely, we launch a preliminary simulated study of seamless remote HR measurement using multiple synchronized cameras by combining ensemble empirical mode decomposition (EEMD) with time-delay canonical correlation analysis (TDCCA), termed as EEMD-TDCCA. At each time point, a target ROI with the largest area is first determined from all the ROIs provided by all the cameras. Then, the RGB time sequence is formed by taking average of all pixels within each target ROI. Afterwards, the green channel time sequence is decomposed into several intrinsic mode functions (IMFs) and only the IMF candidates, whose frequency corresponding to the maximum amplitude falling into the interested HR range will be further processed by TDCCA. Finally, the first pair of the canonical variables having the largest correlation coefficient is the HR source and the corresponding HR is derived by peak detection or frequency analysis. Thirty subjects were recruited and four state-of-the-art methods were employed for comparison. The best performance was achieved by using the proposed EEMD-TDCCA followed by frequency analysis, with the mean absolute error 4.11 bpm, mean percentage error 5.26%, root mean square error 5.37 bpm, the Pearson’s correlation coefficient 0.90 and the intra-class correlation coefficient 0.89, demonstrating the feasibility of our proposed seamless remote HR measurement framework. This study will provide a promising solution for practical and robust non-contact HR measurement applications.

[1]  J. Fleiss.  The design and analysis of clinical experiments , 1987 .

[2]  Z. Jane Wang,et al.  A Novel EEMD-CCA Approach to Removing Muscle Artifacts for Pervasive EEG , 2019, IEEE Sensors Journal.

[3]  L. O. Svaasand,et al.  Remote plethysmographic imaging using ambient light. , 2008, Optics express.

[4]  Gerard de Haan,et al.  Robust Pulse Rate From Chrominance-Based rPPG , 2013, IEEE Transactions on Biomedical Engineering.

[5]  Steffen Leonhardt,et al.  Unobtrusive Vital Sign Monitoring in Automotive Environments—A Review , 2018, Sensors.

[6]  Yu Sun,et al.  Use of ambient light in remote photoplethysmographic systems: comparison between a high-performance camera and a low-cost webcam. , 2012, Journal of biomedical optics.

[7]  Rabab Kreidieh Ward,et al.  A Preliminary Study of Muscular Artifact Cancellation in Single-Channel EEG , 2014, Sensors.

[8]  Rosalind W. Picard,et al.  Non-contact, automated cardiac pulse measurements using video imaging and blind source separation , 2022 .

[9]  L. Tarassenko,et al.  Non-contact video-based vital sign monitoring using ambient light and auto-regressive models , 2014, Physiological measurement.

[10]  Chern-Loon Lim,et al.  Dynamic heart rate estimation using principal component analysis. , 2015, Biomedical optics express.

[11]  Ethan B. Blackford,et al.  Recovering pulse rate during motion artifact with a multi-imager array for non-contact imaging photoplethysmography , 2014, 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[12]  Ethan B. Blackford,et al.  Using consumer-grade devices for multi-imager non-contact imaging photoplethysmography , 2017, BiOS.

[13]  Rabab K. Ward,et al.  Video-Based Heart Rate Measurement: Recent Advances and Future Prospects , 2019, IEEE Transactions on Instrumentation and Measurement.

[14]  Matti Pietikäinen,et al.  Remote Heart Rate Measurement from Face Videos under Realistic Situations , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Hong Liu,et al.  Linear canonical correlation analysis based ranking approach for facial age estimation , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[16]  Asanka G. Perera,et al.  Remote monitoring of cardiorespiratory signals from a hovering unmanned aerial vehicle , 2017, BioMedical Engineering OnLine.

[17]  Sander Stuijk,et al.  A Novel Algorithm for Remote Photoplethysmography: Spatial Subspace Rotation , 2016, IEEE Transactions on Biomedical Engineering.

[18]  Charlotte Soneson,et al.  Integrative analysis of gene expression and copy number alterations using canonical correlation analysis , 2010, BMC Bioinformatics.

[19]  Peter J Lesniewski,et al.  Signal recovery in imaging photoplethysmography , 2013, Physiological measurement.

[20]  Hong Yan,et al.  A Machine Learning Approach to Improve Contactless Heart Rate Monitoring Using a Webcam , 2014, IEEE Journal of Biomedical and Health Informatics.

[21]  Tomasz Kocejko,et al.  Proceedings of the Federated Conference on Computer Science and Information Systems pp. 405–410 ISBN 978-83-60810-22-4 Measuring Pulse Rate with a Webcam – a Non-contact Method for Evaluating Cardiac Activity , 2022 .

[22]  Yu Sun,et al.  Motion-compensated noncontact imaging photoplethysmography to monitor cardiorespiratory status during exercise. , 2011, Journal of biomedical optics.

[23]  Thomas B. Moeslund,et al.  Heartbeat Rate Measurement from Facial Video , 2016, IEEE Intelligent Systems.

[24]  David A. Clausi,et al.  Feasibility of long-distance heart rate monitoring using transmittance photoplethysmographic imaging (PPGI) , 2015, Scientific reports.

[25]  Fang Zhao,et al.  Camera-Based, Non-Contact, Vital-Signs Monitoring Technology May Provide a Way for the Early Prevention of SIDS in Infants , 2016, Front. Neurol..

[26]  Xueyuan Xu,et al.  The Use of Multivariate EMD and CCA for Denoising Muscle Artifacts From Few-Channel EEG Recordings , 2018, IEEE Transactions on Instrumentation and Measurement.

[27]  Kwang Suk Park,et al.  Heart rate estimation from facial photoplethysmography during dynamic illuminance changes , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[28]  Philip H. S. Torr,et al.  Approximate structured output learning for Constrained Local Models with application to real-time facial feature detection and tracking on low-power devices , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[29]  W. Verkruysse,et al.  Non-contact heart rate monitoring utilizing camera photoplethysmography in the neonatal intensive care unit - a pilot study. , 2013, Early human development.

[30]  Lai-Man Po,et al.  Block-based adaptive ROI for remote photoplethysmography , 2018, Multimedia Tools and Applications.

[31]  Lai-Man Po,et al.  Motion-Resistant Remote Imaging Photoplethysmography Based on the Optical Properties of Skin , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[32]  Suh-Yin Lee,et al.  Image Sensor-Based Heart Rate Evaluation From Face Reflectance Using Hilbert–Huang Transform , 2015, IEEE Sensors Journal.

[33]  A S Malik,et al.  Novel health monitoring method using an RGB camera. , 2017, Biomedical optics express.

[34]  Xun Chen,et al.  Illumination Variation-Resistant Video-Based Heart Rate Measurement Using Joint Blind Source Separation and Ensemble Empirical Mode Decomposition , 2017, IEEE Journal of Biomedical and Health Informatics.

[35]  Xun Chen,et al.  A Three-Step Multimodal Analysis Framework for Modeling Corticomuscular Activity With Application to Parkinson’s Disease , 2014, IEEE Journal of Biomedical and Health Informatics.

[36]  Sander Stuijk,et al.  Algorithmic Principles of Remote PPG , 2017, IEEE Transactions on Biomedical Engineering.

[37]  Zhiqi Shen,et al.  Video-based human heart rate measurement using joint blind source separation , 2017, Biomed. Signal Process. Control..

[38]  Giuseppe Polese,et al.  EDCAR: A knowledge representation framework to enhance automatic video surveillance , 2019, Expert Syst. Appl..

[39]  Raymond Chiong,et al.  Remote heart rate measurement using low-cost RGB face video: a technical literature review , 2018, Frontiers of Computer Science.

[40]  Rabab K. Ward,et al.  Removing Muscle Artifacts From EEG Data: Multichannel or Single-Channel Techniques? , 2016, IEEE Sensors Journal.

[41]  Mika P. Tarvainen,et al.  An advanced detrending method with application to HRV analysis , 2002, IEEE Transactions on Biomedical Engineering.

[42]  Magdalena Madej,et al.  Measuring Pulse Rate with a Webcam , 2012 .

[43]  Zdenek Kalal,et al.  Tracking-Learning-Detection , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  Vince D. Calhoun,et al.  Canonical Correlation Analysis for Data Fusion and Group Inferences , 2010, IEEE Signal Processing Magazine.

[45]  Ethan B. Blackford,et al.  Fusing Partial Camera Signals for Noncontact Pulse Rate Variability Measurement , 2018, IEEE Transactions on Biomedical Engineering.

[46]  Juan Cheng,et al.  Illumination variation interference suppression in remote PPG using PLS and MEMD , 2017 .

[47]  H Malberg,et al.  Camera-based photoplethysmography in critical care patients. , 2016, Clinical hemorheology and microcirculation.

[48]  Zhao Yang,et al.  Motion-tolerant heart rate estimation from face videos using derivative filter , 2019, Multimedia Tools and Applications.

[49]  Norden E. Huang,et al.  Ensemble Empirical Mode Decomposition: a Noise-Assisted Data Analysis Method , 2009, Adv. Data Sci. Adapt. Anal..

[50]  Yu Sun,et al.  Photoplethysmography Revisited: From Contact to Noncontact, From Point to Imaging , 2016, IEEE Transactions on Biomedical Engineering.

[51]  Zhiqi Shen,et al.  Physiological parameter monitoring of drivers based on video data and independent vector analysis , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[52]  Xun Chen,et al.  Joint Blind Source Separation for Neurophysiological Data Analysis: Multiset and multimodal methods , 2016, IEEE Signal Processing Magazine.

[53]  N. Huang,et al.  The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis , 1998, Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[54]  Daniel McDuff,et al.  Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam , 2011, IEEE Transactions on Biomedical Engineering.