Non-Contact Physiological Parameters Extraction Using Facial Video Considering Illumination, Motion, Movement and Vibration

Objective: In this paper, four physiological parameters, i.e., heart rate (HR), inter-beat-interval (IBI), heart rate variability (HRV), and oxygen saturation (SpO2), are extracted from facial video recordings. Methods: Facial videos were recorded for 10 min each in 30 test subjects while driving a simulator. Four regions of interest (ROIs) are automatically selected in each facial image frame based on 66 facial landmarks. Red-green-blue color signals are extracted from the ROIs and four physiological parameters are extracted from the color signals. For the evaluation, physiological parameters are also recorded simultaneously using a traditional sensor “cStress,” which is attached to hands and fingers of test subjects. Results: The Bland Altman plots show 95% agreement between the camera system and “cStress” with the highest correlation coefficient R = 0.96 for both HR and SpO2. The quality index is estimated for IBI considering 100 ms R-peak error; the accumulated percentage achieved is 97.5%. HRV features in both time and frequency domains are compared and the highest correlation coefficient achieved is 0.93. One-way analysis of variance test shows that there are no statistically significant differences between the measurements by camera and reference sensors. Conclusion: These results present high degrees of accuracy of HR, IBI, HRV, and SpO2 extraction from facial image sequences. Significance: The proposed non-contact approach could broaden the dimensionality of physiological parameters extraction using cameras. This proposed method could be applied for driver monitoring application under realistic conditions, i.e., illumination, motion, movement, and vibration.

[1]  Stefanos Zafeiriou,et al.  Robust Discriminative Response Map Fitting with Constrained Local Models , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  David Fofi,et al.  Heart rate estimation using facial video: A review , 2017, Biomed. Signal Process. Control..

[3]  Léon J. M. Rothkrantz,et al.  Noncontact automatic heart rate analysis in visible spectrum by specific face regions , 2013, CompSysTech '13.

[4]  Frédéric Bousefsaf,et al.  Remote assessment of physiological parameters by non-contact technologies to quantify and detect mental stress states , 2014, 2014 International Conference on Control, Decision and Information Technologies (CoDIT).

[5]  Ki H. Chon,et al.  Statistical analysis of heart rate and heart rate variability monitoring through the use of smart phone cameras , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[6]  D. Cavouras,et al.  A Simple algorithm to monitor HR for real time treatment applications , 2009, 2009 9th International Conference on Information Technology and Applications in Biomedicine.

[7]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[8]  Hamidur Rahman,et al.  Real Time Heart Rate Monitoring From Facial RGB Color Video Using Webcam , 2016, SAIS.

[9]  Marc Garbey,et al.  Contact-Free Measurement of Cardiac Pulse Based on the Analysis of Thermal Imagery , 2007, IEEE Transactions on Biomedical Engineering.

[10]  Hamidur Rahman,et al.  SmartMirror: An Embedded Non-contact System for Health Monitoring at Home , 2016, HealthyIoT.

[11]  Chunyan Miao,et al.  Non-contact driver cardiac physiological monitoring using video data , 2015, 2015 IEEE China Summit and International Conference on Signal and Information Processing (ChinaSIP).

[12]  Changick Kim,et al.  A robust real time system for remote heart rate measurement via camera , 2015, 2015 IEEE International Conference on Multimedia and Expo (ICME).

[13]  Hamidur Rahman,et al.  Vision-Based Remote Heart Rate Variability Monitoring Using Camera , 2017, HealthyIoT.

[14]  Masayuki Ishihara,et al.  Development of Non-contact Monitoring System of Heart Rate Variability (HRV) - An Approach of Remote Sensing for Ubiquitous Technology - , 2009, HCI.

[15]  Marcel J. T. Reinders,et al.  Image sequence restoration in the presence of pathological motion and severe artifacts , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[16]  J. M. Cho,et al.  A preliminary study on photoplethysmogram (PPG) signal analysis for reduction of motion artifact in frequency domain , 2012, 2012 IEEE-EMBS Conference on Biomedical Engineering and Sciences.

[17]  Murtaza Bulut,et al.  Camera-based heart rate monitoring in highly dynamic light conditions , 2013, 2013 International Conference on Connected Vehicles and Expo (ICCVE).

[18]  Rosalind W. Picard,et al.  Non-contact, automated cardiac pulse measurements using video imaging and blind source separation , 2022 .

[19]  Guy A. Dumont,et al.  Estimation of Respiratory Rate From Photoplethysmographic Imaging Videos Compared to Pulse Oximetry , 2015, IEEE Journal of Biomedical and Health Informatics.

[20]  Matti Pietikäinen,et al.  Remote Heart Rate Measurement from Face Videos under Realistic Situations , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Kevin Hung,et al.  Preliminary investigation of pupil size variability: toward non-contact assessment of cardiovascular variability , 2006, 2006 3rd IEEE/EMBS International Summer School on Medical Devices and Biosensors.

[22]  J M Bland,et al.  Statistical methods for assessing agreement between two methods of clinical measurement , 1986 .

[23]  Yadong Wang,et al.  A review of non-contact, low-cost physiological information measurement based on photoplethysmographic imaging , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[24]  German Da Costa,et al.  Optical remote sensing of heartbeats , 1995 .

[25]  Zhiqi Shen,et al.  Physiological parameter monitoring of drivers based on video data and independent vector analysis , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[26]  Ming Wang,et al.  Webcam based non-contact real-time monitoring for the physiological parameters of drivers , 2014, The 4th Annual IEEE International Conference on Cyber Technology in Automation, Control and Intelligent.

[27]  Ho-Jin Choi,et al.  Motion artifact reduction in PPG signals from face: Face tracking & stochastic state space modeling approach , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[28]  C. Takano,et al.  Heart rate measurement based on a time-lapse image. , 2007, Medical engineering & physics.

[29]  L. O. Svaasand,et al.  Remote plethysmographic imaging using ambient light. , 2008, Optics express.

[30]  Lai-Man Po,et al.  Motion artifacts suppression for remote imaging photoplethysmography , 2014, 2014 19th International Conference on Digital Signal Processing.

[31]  Magdalena Madej,et al.  Measuring Pulse Rate with a Webcam , 2012 .

[32]  Wen Jun Jiang,et al.  Real-time quantifying heart beat rate from facial video recording on a smart phone using Kalman filters , 2014, 2014 IEEE 16th International Conference on e-Health Networking, Applications and Services (Healthcom).

[33]  U. Bal.  Non-contact estimation of heart rate and oxygen saturation using ambient light. , 2015, Biomedical optics express.

[34]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[35]  Ashikur Rahman,et al.  Non-invasive heart rate measuring smartphone applications using on-board cameras: A short survey , 2015, 2015 International Conference on Networking Systems and Security (NSysS).

[36]  Roberto Spinola Barbosa,et al.  Vehicle Vibration Response Subjected to Longwave Measured Pavement Irregularity , 2012 .

[37]  A M Tonkin,et al.  Application of the Poincaré plot to heart rate variability: a new measure of functional status in heart failure. , 1995, Australian and New Zealand journal of medicine.

[38]  Kwang Suk Park,et al.  Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[39]  Yuting Yang,et al.  Noncontact Monitoring of Blood Oxygen Saturation Using Camera and Dual-Wavelength Imaging System , 2016, IEEE Transactions on Biomedical Engineering.

[40]  Ricardo Gutierrez-Osuna,et al.  Contactless Measurement of Heart Rate Variability from Pupillary Fluctuations , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[41]  B. G. Sudarshan,et al.  GUI creation for removal of motion artifact in PPG signals , 2016, 2016 3rd International Conference on Advanced Computing and Communication Systems (ICACCS).

[42]  Hamidur Rahman,et al.  Non-Contact Heart Rate Monitoring Using Lab Color Space , 2016, pHealth.

[43]  Janko Drnovsek,et al.  Non-contact heart rate and heart rate variability measurements: A review , 2014, Biomed. Signal Process. Control..

[44]  Ioannis T. Pavlidis,et al.  StressCam: non-contact measurement of users' emotional states through thermal imaging , 2005, CHI Extended Abstracts.

[45]  Hamidur Rahman,et al.  Quality Index Analysis on Camera-Based R-Peak Identification Considering Movements and Light Illumination , 2018, pHealth.

[46]  Ajoy Kumar,et al.  Performance Comparison of Modified LMS and RLS Algorithms in De- noising of ECG Signals , 2012 .

[47]  Rafael C. González,et al.  Local Determination of a Moving Contrast Edge , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Hamidur Rahman,et al.  Non-contact Physiological Parameters Extraction Using Camera , 2015, IoT 360.

[49]  Daniel McDuff,et al.  Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam , 2011, IEEE Transactions on Biomedical Engineering.

[50]  Kristof Meding,et al.  Automatic detection of motion artifacts in MR images using CNNS , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[51]  T. Ward,et al.  Noncontact simultaneous dual wavelength photoplethysmography: a further step toward noncontact pulse oximetry. , 2007, The Review of scientific instruments.

[52]  Sujay Deb,et al.  Determination of SpO2 and heart-rate using smartphone camera , 2014, Proceedings of The 2014 International Conference on Control, Instrumentation, Energy and Communication (CIEC).

[53]  Yuan-Ting Zhang,et al.  Extraction of Heart Rate Variability from Smartphone Photoplethysmograms , 2015, Comput. Math. Methods Medicine.