Physiological and Affective Computing through Thermal Imaging: A Survey

Thermal imaging-based physiological and affective computing is an emerging research area enabling technologies to monitor our bodily functions and understand psychological and affective needs in a contactless manner. However, up to recently, research has been mainly carried out in very controlled lab settings. As small size and even low-cost versions of thermal video cameras have started to appear on the market, mobile thermal imaging is opening its door to ubiquitous and real-world applications. Here we review the literature on the use of thermal imaging to track changes in physiological cues relevant to affective computing and the technological requirements set so far. In doing so, we aim to establish computational and methodological pipelines from thermal images of the human skin to affective states and outline the research opportunities and challenges to be tackled to make ubiquitous real-life thermal imaging-based affect monitoring a possibility.

[1]  P. Lang.  International affective picture system (IAPS) : affective ratings of pictures and instruction manual , 2005 .

[2]  E. Ring,et al.  Infrared thermal imaging in medicine , 2012, Physiological measurement.

[3]  Gary G Berntson,et al.  A non-contact technique for measuring eccrine sweat gland activity using passive thermal imaging. , 2014, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[4]  Amin Mohammadian,et al.  Spatio-temporal filtering of thermal video sequences for heart rate estimation , 2016, Expert Syst. Appl..

[5]  James R. Jackson.  Case of a Cartilaginous Tumor in the Trachea, Producing Dyspnœa, &c. , 1828 .

[6]  Burcin Becerik-Gerber,et al.  Towards unsupervised learning of thermal comfort using infrared thermography , 2018 .

[7]  L. O. Svaasand,et al.  Remote plethysmographic imaging using ambient light. , 2008, Optics express.

[8]  Nicolai Marquardt,et al.  Robust tracking of respiratory rate in high-dynamic range scenes using mobile thermal imaging. , 2017, Biomedical optics express.

[9]  B. Wallin,et al.  Skin blood flow responses to mental stress in man depend on body temperature. , 1987, Acta physiologica Scandinavica.

[10]  Nicolai Marquardt,et al.  Deep Thermal Imaging: Proximate Material Type Recognition in the Wild through Deep Learning of Spatial Surface Temperature Patterns , 2018, CHI.

[11]  Frank Birklein,et al.  Complex regional pain syndrome , 2005, Journal of Neurology.

[12]  Luis Alberto Morales Hernández,et al.  Biopsychosocial Assessment of Pain with Thermal Imaging of Emotional Facial Expression in Breast Cancer Survivors , 2018 .

[13]  Shuchang Xu,et al.  Robust efficient estimation of heart rate pulse from video. , 2014, Biomedical optics express.

[14]  Nadia Bianchi-Berthouze,et al.  Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging , 2018, bioRxiv.

[15]  Aly A. Farag,et al.  A Fully Automatic Method to Extract the Heart Rate from Thermal Video , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[16]  Marc Garbey,et al.  Contact-Free Measurement of Cardiac Pulse Based on the Analysis of Thermal Imagery , 2007, IEEE Transactions on Biomedical Engineering.

[17]  P. Ekman.  Facial expression and emotion. , 1993, The American psychologist.

[18]  Sander Stuijk,et al.  Robust respiration detection from remote photoplethysmography. , 2016, Biomedical optics express.

[19]  Arcangelo Merla,et al.  Exploring the Use of Thermal Infrared Imaging in Human Stress Research , 2014, PloS one.

[20]  Sheng Hong,et al.  Real-time stress assessment using thermal imaging , 2016, The Visual Computer.

[21]  Vittorio Gallese,et al.  Seeing a Blush on the Visible and Invisible Spectrum: A Functional Thermal Infrared Imaging Study , 2017, Front. Hum. Neurosci..

[22]  Steffen Leonhardt,et al.  Remote monitoring of breathing dynamics using infrared thermography. , 2015, Biomedical optics express.

[23]  F. Shaffer,et al.  An Overview of Heart Rate Variability Metrics and Norms , 2017, Front. Public Health.

[24]  Alan Chalmers,et al.  Evaluation of tone mapping operators using a High Dynamic Range display , 2005, ACM Trans. Graph..

[25]  L. H. Ziegler,et al.  A STUDY OF THE INFLUENCE OF EMOTIONS AND AFFECTS ON THE SURFACE TEMPERATURE OF THE HUMAN BODY , 1938 .

[26]  Nicu Sebe,et al.  Facial expression recognition from video sequences: temporal and static modeling , 2003, Comput. Vis. Image Underst..

[27]  Vittorio Gallese,et al.  Sympathy Crying: Insights from Infrared Thermal Imaging on a Female Sample , 2016, PloS one.

[28]  Jin-Hyuk Hong,et al.  Understanding physiological responses to stressors during physical activity , 2012, UbiComp.

[29]  Michael Isard,et al.  CONDENSATION—Conditional Density Propagation for Visual Tracking , 1998, International Journal of Computer Vision.

[30]  Thomas S. Huang,et al.  Explanation-based facial motion tracking using a piecewise Bezier volume deformation model , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[31]  Yan Zhou,et al.  Spatiotemporal Smoothing as a Basis for Facial Tissue Tracking in Thermal Imaging , 2013, IEEE Transactions on Biomedical Engineering.

[32]  Daniel McDuff,et al.  Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam , 2011, IEEE Transactions on Biomedical Engineering.

[33]  David Sander,et al.  Thermal Analysis of Facial Muscles Contractions , 2011, IEEE Transactions on Affective Computing.

[34]  Pradeep Buddharaju,et al.  On enhancing cardiac pulse measurements through thermal imaging , 2009, 2009 9th International Conference on Information Technology and Applications in Biomedicine.

[35]  Sidney K. D'Mello,et al.  Time to Scale: Generalizable Affect Detection for Tens of Thousands of Students across An Entire School Year , 2019, CHI.

[36]  John E. Hall,et al.  Guyton and Hall Textbook of Medical Physiology , 2015 .

[37]  Tom Chau,et al.  Classifying Affective States Using Thermal Infrared Imaging of the Human Face , 2010, IEEE Transactions on Biomedical Engineering.

[38]  I. Pavlidis,et al.  Touchless monitoring of breathing function , 2004, The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[39]  J. M. Lloyd,et al.  Thermal Imaging Systems , 1975 .

[40]  Ioannis T. Pavlidis,et al.  Thermal image analysis for anxiety detection , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[41]  A. Merla,et al.  Mother and child in synchrony: Thermal facial imprints of autonomic contagion , 2012, Biological Psychology.

[42]  R GRANT,et al.  Emotional hypothermia in rabbits. , 1950, The American journal of physiology.

[43]  Gert R. G. Lanckriet,et al.  A random forest classifier for the prediction of energy expenditure and type of physical activity from wrist and hip accelerometers , 2014, Physiological measurement.

[44]  Dvijesh Shastri,et al.  Imaging Facial Physiology for the Detection of Deceit , 2006, International Journal of Computer Vision.

[45]  E. Salazar-López,et al.  The mental and subjective skin: Emotion, empathy, feelings and thermography , 2015, Consciousness and Cognition.

[46]  Adrian Basarab,et al.  Towards an automatic early stress recognition system for office environments based on multimodal measurements: A review , 2016, J. Biomed. Informatics.

[47]  Edward H. Adelson,et al.  Exploring features in a Bayesian framework for material recognition , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[48]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[49]  Shangfei Wang,et al.  Emotion Recognition Using Hidden Markov Models from Facial Temperature Sequence , 2011, ACII.

[50]  Ioannis T. Pavlidis,et al.  Thermistor at a Distance: Unobtrusive Measurement of Breathing , 2010, IEEE Transactions on Biomedical Engineering.

[51]  Arcangelo Merla,et al.  Thermal Infrared Imaging-Based Computational Psychophysiology for Psychometrics , 2015, Comput. Math. Methods Medicine.

[52]  Amanda C. Hahn,et al.  Hot or not? Thermal reactions to social contact , 2012, Biology Letters.

[53]  Filippo Zappasodi,et al.  Autonomic impairment in Alzheimer’s disease is revealed by complexity analysis of functional thermal imaging signals during cognitive tasks , 2019, Physiological measurement.

[54]  Fei Chen,et al.  A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference , 2010, IEEE Transactions on Multimedia.

[55]  S. N. Sidek,et al.  Thermal imaging based affective state recognition , 2015, 2015 IEEE International Symposium on Robotics and Intelligent Sensors (IRIS).

[56]  Dvijesh Shastri,et al.  Imaging Facial Signs of Neurophysiological Responses , 2009, IEEE Transactions on Biomedical Engineering.

[57]  Andrew H. Ryan,et al.  Facial Skin Surface Temperature Changes During a “Concealed Information” Test , 2006, Annals of Biomedical Engineering.

[58]  Ioannis T. Pavlidis,et al.  Coalitional tracking , 2007, Comput. Vis. Image Underst..

[59]  Abbas K. Abbas,et al.  Neonatal non-contact respiratory monitoring based on real-time infrared thermography , 2011, Biomedical engineering online.

[60]  Jennifer Healey,et al.  Detecting stress during real-world driving tasks using physiological sensors , 2005, IEEE Transactions on Intelligent Transportation Systems.

[61]  F. Vetere,et al.  Cognitive Heat , 2017 .

[62]  Albrecht Schmidt,et al.  Ubiquitous computing - computing in context , 2003 .

[63]  Shaogang Gong,et al.  Robust facial expression recognition using local binary patterns , 2005, IEEE International Conference on Image Processing 2005.

[64]  C. Kirschbaum,et al.  The 'Trier Social Stress Test'--a tool for investigating psychobiological stress responses in a laboratory setting. , 1993, Neuropsychobiology.

[65]  Haibin Ling,et al.  Robust Visual Tracking and Vehicle Classification via Sparse Representation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[66]  Zhi Yan,et al.  Thermal Camera Based Physiological Monitoring with an Assistive Robot , 2018, 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[67]  Ioannis Pavlidis,et al.  Seeing through the face of deception , 2002, Nature.

[68]  Masood Mehmood Khan,et al.  Toward Use of Facial Thermal Features in Dynamic Assessment of Affect and Arousal Level , 2017, IEEE Transactions on Affective Computing.

[69]  A. Merla,et al.  Thermal signature of fear conditioning in mild post traumatic stress disorder , 2014, Neuroscience.

[70]  Daniel Huber,et al.  Three-dimensional and thermal surface imaging produces reliable measures of joint shape and temperature: a potential tool for quantifying arthritis , 2008, Arthritis research & therapy.

[71]  D L Kellogg,et al.  Role of sympathetic nerves in the vascular effects of local temperature in human forearm skin. , 1993, The American journal of physiology.

[72]  J S Haller.  Medical thermometry--a short history. , 1985, The Western journal of medicine.

[73]  Simon J. Julier,et al.  Nose Heat: Exploring Stress-induced Nasal Thermal Variability through Mobile Thermal Imaging , 2019, 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII).

[74]  H Donkin,et al.  On Some Cases of Abnormally High Temperature* , 1879, British medical journal.

[75]  Verónica Pérez-Rosas,et al.  Using Infrared Thermography and Biosensors to Detect Thermal Discomfort in a Building’s Inhabitants , 2014 .

[76]  Saho Ayabe-Kanamura,et al.  The Invisible Potential of Facial Electromyography: A Comparison of EMG and Computer Vision when Distinguishing Posed from Spontaneous Smiles , 2019, CHI.

[77]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[78]  Gregory F. Lewis,et al.  A novel method for extracting respiration rate and relative tidal volume from infrared thermography. , 2011, Psychophysiology.

[79]  A. Kistler,et al.  Fingertip temperature as an indicator for sympathetic responses. , 1998, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[80]  Seong G. Kong,et al.  Fusion of Visual and Thermal Signatures with Eyeglass Removal for Robust Face Recognition , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[81]  Katsuki Nakamura,et al.  The use of nasal skin temperature measurements in studying emotion in macaque monkeys , 2011, Physiology & Behavior.

[82]  Jaakko Lehtinen,et al.  Two-shot SVBRDF capture for stationary materials , 2015, ACM Trans. Graph..

[83]  Leonardo Trujillo,et al.  Visual learning of texture descriptors for facial expression recognition in thermal imagery , 2007, Comput. Vis. Image Underst..

[84]  M. Cabanac.  What is emotion? , 2002, Behavioural Processes.

[85]  Shunji Goto,et al.  Decrease in nasal temperature of rhesus monkeys (Macaca mulatta) in negative emotional state , 2005, Physiology & Behavior.

[86]  J. Stroop.  Studies of interference in serial verbal reactions. , 1992 .

[87]  Nicolai Marquardt,et al.  ThermSense: Smartphone-based breathing sensing platform using noncontact low-cost thermal camera , 2017, 2017 Seventh International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW).

[88]  Ali Nahvi,et al.  Monitoring the Variation in Driver Respiration Rate from Wakefulness to Drowsiness: A Non-Intrusive Method for Drowsiness Detection Using Thermal Imaging , 2018 .

[89]  G. Wasner,et al.  Skin temperature side differences – a diagnostic tool for CRPS? , 2002, Pain.

[90]  J. S. Lombard.  VI. Experimental researches on the temperature of the head , 1878, Proceedings of the Royal Society of London.

[91]  S. Kornblum,et al.  Isolation of Specific Interference Processing in the Stroop Task: PET Activation Studies , 1997, NeuroImage.

[92]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[93]  I. Pavlidis,et al.  Fast by Nature - How Stress Patterns Define Human Experience and Performance in Dexterous Tasks , 2012, Scientific Reports.

[94]  Leandro L Di Stasi,et al.  Nasal skin temperature reveals changes in arousal levels due to time on task: An experimental thermal infrared imaging study. , 2019, Applied ergonomics.

[95]  Tom Chau,et al.  Thermal Imaging of the Periorbital Regions during the Presentation of an Auditory Startle Stimulus , 2011, PloS one.

[96]  Denis Delisle-Rodriguez,et al.  Emotion analysis in children through facial emissivity of infrared thermal imaging , 2019, PloS one.

[97]  George S. Everly,et al.  A Clinical Guide to the Treatment of the Human Stress Response , 2002 .

[98]  Salvatore Maria Aglioti,et al.  Cognitive load and emotional processing in psoriasis: a thermal imaging study , 2018, Experimental Brain Research.

[99]  Dvijesh Shastri,et al.  Perinasal Imaging of Physiological Stress and Its Affective Potential , 2012, IEEE Transactions on Affective Computing.

[100]  Javier Hernandez,et al.  Call Center Stress Recognition with Person-Specific Models , 2011, ACII.

[101]  Chris Harrison,et al.  OmniTouch: wearable multitouch interaction everywhere , 2011, UIST.

[102]  J. A. Veltman,et al.  Facial temperature as a measure of mental workload , 2005 .

[103]  Yoshihisa Fujiwara,et al.  Using facial skin temperature to objectively evaluate sensations , 1997 .

[104]  Gianluca Esposito,et al.  Using infrared thermography to assess emotional responses to infants , 2015, Early child development and care.

[105]  J. S. Lombard,et al.  Experimental Researches on the Temperature of the Head , 1881, Proceedings of the Royal Society of London.

[106]  Simon J. Julier,et al.  DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings , 2017, 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII).

[107]  P. Grossman.  Respiration, stress, and cardiovascular function. , 1983, Psychophysiology.

[108]  I. Pavlidis,et al.  Noncontact measurement of breathing function , 2006, IEEE Engineering in Medicine and Biology Magazine.

[109]  Ioannis T. Pavlidis,et al.  StressCam: non-contact measurement of users' emotional states through thermal imaging , 2005, CHI Extended Abstracts.

[110]  P. Ekman,et al.  Facial action coding system , 2019 .

[111]  John Pearson.  Observations and Queries on Animal Heat , 1786, The London medical journal.

[112]  Saurabh Singh,et al.  Face recognition by fusing thermal infrared and visible imagery , 2006, Image Vis. Comput..

[113]  Vittorio Gallese,et al.  The Autonomic Signature of Guilt in Children: A Thermal Infrared Imaging Study , 2013, PloS one.

[114]  Vincent G. Duffy,et al.  Development of a facial skin temperature-based methodology for non-intrusive mental workload measurement , 2007 .

[115]  Ricardo Gutierrez-Osuna,et al.  Email Makes You Sweat: Examining Email Interruptions and Stress Using Thermal Imaging , 2019, CHI.

[116]  B. Wallin.  Sympathetic nerve activity underlying electrodermal and cardiovascular reactions in man. , 1981, Psychophysiology.

[117]  Pradeep Buddharaju,et al.  A Comparative Analysis of Thermal and Visual Modalities for Automated Facial Expression Recognition , 2012, ISVC.

[118]  Daniel McDuff,et al.  COGCAM: Contact-free Measurement of Cognitive Stress During Computer Tasks with a Digital Camera , 2016, CHI.

[119]  周鑫,et al.  Tracking-learning-detection (TLD)-based video object tracking method , 2012 .