Towards unravelling the relationship between on-body, environmental and emotion data using sensor information fusion approach

Abstract Over the past few years, there has been a noticeable advancement in environmental models and information fusion systems taking advantage of the recent developments in sensor and mobile technologies. However, little attention has been paid so far to quantifying the relationship between environment changes and their impact on our bodies in real-life settings. In this paper, we identify a data driven approach based on direct and continuous sensor data to assess the impact of the surrounding environment and physiological changes and emotion. We aim at investigating the potential of fusing on-body physiological signals, environmental sensory data and on-line self-report emotion measures in order to achieve the following objectives: (1) model the short term impact of the ambient environment on human body, (2) predict emotions based on-body sensors and environmental data. To achieve this, we have conducted a real-world study ‘in the wild’ with on-body and mobile sensors. Data was collected from participants walking around Nottingham city centre, in order to develop analytical and predictive models. Multiple regression, after allowing for possible confounders, showed a noticeable correlation between noise exposure and heart rate. Similarly, UV and environmental noise have been shown to have a noticeable effect on changes in ElectroDermal Activity (EDA ). Air pressure demonstrated the greatest contribution towards the detected changes in body temperature and motion. Also, significant correlation was found between air pressure and heart rate . Finally, decision fusion of the classification results from different modalities is performed. To the best of our knowledge this work presents the first attempt at fusing and modelling data from environmental and physiological sources collected from sensors in a real-world setting.

[1]  Wan-Young Chung,et al.  A Fusion Health Monitoring Using ECG and Accelerometer sensors for Elderly Persons at Home , 2007, 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[2]  Mariano Alcañiz Raya,et al.  A Comparison of Physiological Signal Analysis Techniques and Classifiers for Automatic Emotional Evaluation of Audiovisual Contents , 2016, Front. Comput. Neurosci..

[3]  K. Takahashi,et al.  Remarks on SVM-based emotion recognition from multi-modal bio-potential signals , 2004, RO-MAN 2004. 13th IEEE International Workshop on Robot and Human Interactive Communication (IEEE Catalog No.04TH8759).

[4]  Albert-László Barabási,et al.  The origin of bursts and heavy tails in human dynamics , 2005, Nature.

[5]  Marimuthu Palaniswami,et al.  Poincaré Plot Methods for Heart Rate Variability Analysis , 2013, Springer US.

[6]  Holger R. Maier,et al.  Integrating modelling and smart sensors for environmental and human health , 2015, Environ. Model. Softw..

[7]  P. Amouyel,et al.  Unhealthy effects of atmospheric temperature and pressure on the occurrence of myocardial infarction and coronary deaths. A 10-year survey: the Lille-World Health Organization MONICA project (Monitoring trends and determinants in cardiovascular disease). , 1999, Circulation.

[8]  David Roberts,et al.  MobSens: Making Smart Phones Smarter , 2009, IEEE Pervasive Computing.

[9]  Andrea Castelletti,et al.  An evaluation framework for input variable selection algorithms for environmental data-driven models , 2014, Environ. Model. Softw..

[10]  Eman M. G. Younis.  Sentiment Analysis and Text Mining for Social Media Microblogs using Open Source Tools: An Empirical Study , 2015 .

[11]  M. Basner,et al.  Cardiovascular effects of environmental noise exposure , 2014, European heart journal.

[12]  Alain Pruski,et al.  Multiresolution framework for emotion sensing in physiological signals , 2016, 2016 2nd International Conference on Advanced Technologies for Signal and Image Processing (ATSIP).

[13]  Wen Wan-Hui,et al.  Electrocardiography Recording, Feature Extraction and Classification for Emotion Recognition , 2009, 2009 WRI World Congress on Computer Science and Information Engineering.

[14]  R. Martin,et al.  Toward the next generation of air quality monitoring: Particulate Matter , 2013 .

[15]  Roozbeh Jafari,et al.  Enabling Effective Programming and Flexible Management of Efficient Body Sensor Network Applications , 2013, IEEE Transactions on Human-Machine Systems.

[16]  Ulrike Groemping,et al.  Relative Importance for Linear Regression in R: The Package relaimpo , 2006 .

[17]  Giancarlo Fortino,et al.  Automatic Methods for the Detection of Accelerative Cardiac Defense Response , 2016, IEEE Transactions on Affective Computing.

[18]  Steve Benford,et al.  Understanding mass participatory pervasive computing systems for environmental campaigns , 2013, Personal and Ubiquitous Computing.

[19]  Alan Chamberlain,et al.  Sense of space: mapping physiological emotion response in urban space , 2013, UbiComp.

[20]  D. Dockery,et al.  Acute respiratory effects of particulate air pollution. , 1994, Annual review of public health.

[21]  Bertha A Hidalgo,et al.  Multivariate or multivariable regression? , 2013, American journal of public health.

[22]  R. B. Knapp,et al.  Physiological signals and their use in augmenting emotion recognition for human-machine interaction , 2011 .

[23]  Péter Apor,et al.  [Heart rate variability analysis in sports]. , 2009, Orvosi hetilap.

[24]  S. Stansfeld,et al.  Noise and Health in the Urban Environment , 2000, Reviews on environmental health.

[25]  Steve Benford,et al.  MobGeoSen: facilitating personal geosensor data collection and visualization using mobile phones , 2007, Personal and Ubiquitous Computing.

[26]  Elisabeth André,et al.  Emotion recognition based on physiological changes in music listening , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Alan Chamberlain,et al.  Shopmobia: An Emotion-Based Shop Rating System , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[28]  N. Verberkmoes,et al.  Impact of temperature and atmospheric pressure on the incidence of major acute cardiovascular events , 2012, Netherlands Heart Journal.

[29]  Abbes Amira,et al.  Emotion Recognition by Physiological Signals , 2016, HVEI.

[30]  K. H. Kim,et al.  Emotion recognition system using short-term monitoring of physiological signals , 2004, Medical and Biological Engineering and Computing.

[31]  L. Rothkrantz.  Multimodal recognition of emotions in car environments , 2009 .

[32]  H. Boshuizen,et al.  The association between noise exposure and blood pressure and ischemic heart disease: a meta-analysis. , 2002, Environmental health perspectives.

[33]  Uwe Schlink,et al.  A conceptual framework for integrated analysis of environmental quality and quality of life , 2014 .

[34]  Lulwah Al-Barrak,et al.  NeuroPlace: making sense of a place , 2013, AH.

[35]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  David L. Smith,et al.  Quantifying the Impact of Human Mobility on Malaria , 2012, Science.

[37]  Sanghyuk Bae,et al.  Air Pollution and Symptoms of Depression in Elderly Adults , 2012, Environmental health perspectives.

[38]  Sylvia D. Kreibig,et al.  Autonomic nervous system activity in emotion: A review , 2010, Biological Psychology.

[39]  Fabien Ringeval,et al.  AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge , 2016, AVEC@ACM Multimedia.

[40]  R. Burnett,et al.  Living near major roads and the incidence of dementia, Parkinson's disease, and multiple sclerosis: a population-based cohort study , 2017, The Lancet.

[41]  Ralf Dörner,et al.  Affective Computing in Games , 2015, Entertainment Computing and Serious Games.

[42]  Athanasios V. Vasilakos,et al.  Body Area Networks: A Survey , 2010, Mob. Networks Appl..

[43]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[44]  Yoav Freund,et al.  A Short Introduction to Boosting , 1999 .

[45]  Alexandra Schneider,et al.  Individual Daytime Noise Exposure during Routine Activities and Heart Rate Variability in Adults: A Repeated Measures Study , 2013, Environmental health perspectives.

[46]  Eiman Kanjo,et al.  A supermarket stress map , 2013, UbiComp.

[47]  J. Allik,et al.  The influence of the weather on affective experience: An experience sampling study. , 2011 .

[48]  Alan Chamberlain,et al.  Emotions in context: examining pervasive affective sensing systems, applications, and analyses , 2015, Personal and Ubiquitous Computing.

[49]  Hassan Ghasemzadeh,et al.  Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges , 2017, Inf. Fusion.

[50]  Zoltan-Csaba Marton,et al.  Ensembles of strong learners for multi-cue classification , 2013, Pattern Recognit. Lett..

[51]  Nikunj C. Oza,et al.  Online Ensemble Learning , 2000, AAAI/IAAI.

[52]  Christine L. Lisetti,et al.  Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals , 2004, EURASIP J. Adv. Signal Process..

[53]  Audrey de Nazelle,et al.  Improving estimates of air pollution exposure through ubiquitous sensing technologies. , 2013, Environmental pollution.

[54]  Martin Schmitz,et al.  Scalable Predictive Analysis in Critically Ill Patients Using a Visual Open Data Analysis Platform , 2016, PloS one.

[55]  SandrineDanet,et al.  Unhealthy Effects of Atmospheric Temperature and Pressure on the Occurrence of Myocardial Infarction and Coronary Deaths , 1999 .

[56]  M. Kampa,et al.  Human health effects of air pollution. , 2008, Environmental pollution.

[57]  Thomas Lange,et al.  Evaluation of human mobility models, for exposure to air pollutants. , 2010, The Science of the total environment.

[58]  David Briggs,et al.  Environmental pollution and the global burden of disease. , 2003, British medical bulletin.

[59]  Alan D. Lopez,et al.  A comparative risk assessment of burden of disease and injury attributable to 67 risk factors and risk factor clusters in 21 regions, 1990–2010: a systematic analysis for the Global Burden of Disease Study 2010 , 2012, The Lancet.

[60]  Sheikh Iqbal Ahamed,et al.  In situ affect detection in mobile devices: a multimodal approach for advertisement using social network , 2013, SIAP.

[61]  Chun Lin,et al.  Personal exposure monitoring of PM2.5 in indoor and outdoor microenvironments. , 2015, The Science of the total environment.

[62]  Hauke Egermann,et al.  From Motion to Emotion: Accelerometer Data Predict Subjective Experience of Music , 2016, PloS one.

[63]  Cheryl Farr,et al.  The Effects of Lighting on Consumers’ Emotions and Behavioral Intentions in a Retail Environment: A Cross–Cultural Comparison , 2007 .

[64]  Eman M. G. Younis,et al.  NeuroPlace: Categorizing urban places according to mental states , 2017, PloS one.

[65]  C. Sabel,et al.  Quantifying human exposure to air pollution--moving from static monitoring to spatio-temporally resolved personal exposure assessment. , 2013, The Science of the total environment.