Extracting Heart Rate from Videos of Online Participants

Crowdsourcing experiments online allows for low-cost data gathering with large participant pools; however, collecting data online does not give researchers access to certain metrics. For example, physiological measures such as heart rate (HR) can provide high-resolution data about the physical, emotional, and mental state of the participant. We investigate and characterize the feasibility of gathering HR from videos of online participants engaged in single user and social tasks. We show that room lighting, head motion, and network bandwidth influence measurement quality, but that instructing participants in good practices substantially improves measurement quality. Our work takes a step towards online physiological data collection.

[1]  Regan L. Mandryk,et al.  Using psychophysiological techniques to measure user experience with entertainment technologies , 2006, Behav. Inf. Technol..

[2]  Sander Stuijk,et al.  Exploiting Spatial Redundancy of Image Sensor for Motion Robust rPPG , 2015, IEEE Transactions on Biomedical Engineering.

[3]  J. Rutenfranz,et al.  Physical performance capacity of children in norway , 2004, European Journal of Applied Physiology and Occupational Physiology.

[4]  L. T. DeCarlo.  On the meaning and use of kurtosis. , 1997 .

[5]  A. K. Blangsted,et al.  The effect of mental stress on heart rate variability and blood pressure during computer work , 2004, European Journal of Applied Physiology.

[6]  B. Kudielka,et al.  Acute HPA axis responses, heart rate, and mood changes to psychosocial stress (TSST) in humans at different times of day , 2004, Psychoneuroendocrinology.

[7]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[8]  E. Hess,et al.  Pupil Size in Relation to Mental Activity during Simple Problem-Solving , 1964, Science.

[9]  Regan L. Mandryk,et al.  A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies , 2007, Int. J. Hum. Comput. Stud..

[10]  Lennart E. Nacke,et al.  Games User Research and Physiological Game Evaluation , 2015, Game User Experience Evaluation.

[11]  Aniket Kittur,et al.  CrowdForge: crowdsourcing complex work , 2011, UIST.

[12]  B. J. Casey,et al.  Heart rate variability during attention phases in young infants. , 1991, Psychophysiology.

[13]  Katherine Isbister,et al.  Game Usability - Advice from the Experts for Advancing the Player Experience , 2008 .

[14]  Karel Brookhuis,et al.  Estimating mental effort using heart rate and heart rate variability. , 2004 .

[15]  Regan L. Mandryk,et al.  Identifying emotional states using keystroke dynamics , 2011, CHI.

[16]  P. Åstrand,et al.  A nomogram for calculation of aerobic capacity (physical fitness) from pulse rate during sub-maximal work. , 1954, Journal of applied physiology.

[17]  John J. B. Allen,et al.  The handbook of emotion elicitation and assessment , 2007 .

[18]  Andreas Krause,et al.  Robust, low-cost, non-intrusive sensing and recognition of seated postures , 2007, UIST.

[19]  K. Lange Andersen,et al.  Physical performance capacity of children in Norway , 2004, European Journal of Applied Physiology and Occupational Physiology.

[20]  Georgios N. Yannakakis,et al.  Correlation between heart rate, electrodermal activity and player experience in first-person shooter games , 2010, Sandbox '10.

[21]  Karel Brookhuis,et al.  Handbook of Human Factors and Ergonomics Methods , 2009 .

[22]  Rosalind W. Picard,et al.  Non-contact, automated cardiac pulse measurements using video imaging and blind source separation , 2022 .

[23]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[24]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[25]  Hagen Malberg,et al.  Heart beat detection and analysis from videos , 2014, 2014 IEEE 34th International Scientific Conference on Electronics and Nanotechnology (ELNANO).

[26]  Richard J. Duro,et al.  Digital Image and Signal Processing for Measurement Systems , 2012 .

[27]  L. O. Svaasand,et al.  Remote plethysmographic imaging using ambient light. , 2008, Optics express.

[28]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[29]  Amar Cheema,et al.  Data collection in a flat world: the strengths and weaknesses of mechanical turk samples , 2013 .

[30]  Yu Sun,et al.  Motion-compensated noncontact imaging photoplethysmography to monitor cardiorespiratory status during exercise. , 2011, Journal of biomedical optics.

[31]  Brent Simpson,et al.  Emotional reactions to losing explain gender differences in entering a risky lottery , 2010, Judgment and Decision Making.

[32]  Eric Gilbert,et al.  Comparing Person- and Process-centric Strategies for Obtaining Quality Data on Amazon Mechanical Turk , 2015, CHI.

[33]  J Aasman,et al.  Operator Effort and the Measurement of Heart-Rate Variability , 1987, Human factors.

[34]  J. Cacioppo,et al.  Handbook Of Psychophysiology , 2019 .

[35]  T. Pursche,et al.  Video-based heart rate measurement from human faces , 2012, 2012 IEEE International Conference on Consumer Electronics (ICCE).