Towards pervasive geospatial affect perception

Due to the enormous penetration of connected computing devices with diverse sensing and localization capabilities, a good fraction of an individual’s activities, locations, and social connections can be sensed and spatially pinpointed. We see significant potential to advance the field of personal activity sensing and tracking beyond its current state of simple activities, at the same time linking activities geospatially. We investigate the detection of sentiment from environmental, on-body and smartphone sensors and propose an affect map as an interface to accumulate and interpret data about emotion and mood from diverse set of sensing sources. In this paper, we first survey existing work on affect sensing and geospatial systems, before presenting a taxonomy of large-scale affect sensing. We discuss model relationships among human emotions and geo-spaces using networks, apply clustering algorithms to the networks and visualize clusters on a map considering space, time and mobility. For the recognition of emotion and mood, we report from two studies exploiting environmental and on-body sensors. Thereafter, we propose a framework for large-scale affect sensing and discuss challenges and open issues for future work.

[1]  Yoshihide Sekimoto,et al.  PFlow: Reconstructing People Flow Recycling Large-Scale Social Survey Data , 2011, IEEE Pervasive Computing.

[2]  T. Dalgleish,et al.  Handbook of Cognition and Emotion: Dalgleish/Cognition and Emotion , 2005 .

[3]  A. Mehrabian Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in Temperament , 1996 .

[4]  Wendy S. Ark,et al.  The Emotion Mouse , 1999, HCI.

[5]  Shashidhar G. Koolagudi,et al.  Emotion recognition from speech: a review , 2012, International Journal of Speech Technology.

[6]  Gentiane Venture,et al.  Recognizing Emotions Conveyed by Human Gait , 2014, International Journal of Social Robotics.

[7]  K. Scherer What are emotions? And how can they be measured? , 2005 .

[8]  M. Bradley,et al.  Looking at pictures: affective, facial, visceral, and behavioral reactions. , 1993, Psychophysiology.

[9]  Khaled A. Harras,et al.  Wigest: A Ubiquitous Wifi-based Gesture Recognition System , 2014 .

[10]  J. Russell A circumplex model of affect. , 1980 .

[11]  Stephan Sigg,et al.  Applicability of RF-based methods for emotion recognition: A survey , 2016, 2016 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops).

[12]  Robert C. Wolpert,et al.  A Review of the , 1985 .

[13]  Hosub Lee,et al.  Towards unobtrusive emotion recognition for affective social communication , 2012, 2012 IEEE Consumer Communications and Networking Conference (CCNC).

[14]  A. Grob,et al.  Dimensional models of core affect: a quantitative comparison by means of structural equation modeling , 2000 .

[15]  Anja Bachmann,et al.  Leveraging smartwatches for unobtrusive mobile ambulatory mood assessment , 2015, UbiComp/ISWC Adjunct.

[16]  Brian R. Baucom,et al.  Startle modulation before, during and after exposure to emotional stimuli. , 2002, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[17]  Antonio Corradi,et al.  Crowdsensing in Urban Areas for City-Scale Mass Gathering Management: Geofencing and Activity Recognition , 2014, IEEE Sensors Journal.

[18]  Elizabeth A. Crane,et al.  Motion Capture and Emotion: Affect Detection in Whole Body Movement , 2007, ACII.

[19]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[20]  B. de Gelder Why bodies? Twelve reasons for including bodily expressions in affective neuroscience. , 2009, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[21]  P. Terry,et al.  Development and validation of the emotion and mood components of Anxiety Questionnaire , 2005 .

[22]  Peter J. Lang,et al.  Gaze Patterns When Looking at Emotional Pictures: Motivationally Biased Attention , 2004 .

[23]  Daniel McDuff,et al.  Biophone: Physiology monitoring from peripheral smartphone motions , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[24]  Tom Geller How do you feel?: your computer knows , 2014, CACM.

[25]  Tiago H. Moreira de Oliveira,et al.  The emotion-aware city: using ambient geographic information (AGI) in order to understand emotion & stress within smart cities , 2015, AGILE PhD School.

[26]  Michael Beigl,et al.  A wearable system for mood assessment considering smartphone features and data from mobile ECGs , 2016, UbiComp Adjunct.

[27]  P. Ekman Are there basic emotions? , 1992, Psychological review.

[28]  Luca Citi,et al.  Revealing Real-Time Emotional Responses: a Personalized Assessment based on Heartbeat Dynamics , 2014, Scientific Reports.

[29]  Robert F. Stanners,et al.  The pupillary response as an indicator of arousal and cognition , 1979 .

[30]  Sazali Yaacob,et al.  FCM clustering of emotional stress using ECG features , 2013, 2013 International Conference on Communication and Signal Processing.

[31]  E. Granholm,et al.  Pupillometric measures of cognitive and emotional processes. , 2004, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[32]  Gerhard Tröster,et al.  The telepathic phone: Frictionless activity recognition from WiFi-RSSI , 2014, 2014 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[33]  Panagiota Anastasopoulou,et al.  Mobile monitoring of epileptic patients using a reconfigurable cyberphysical system that handles multi-parametric data acquisition and analysis , 2014, 2014 4th International Conference on Wireless Mobile Communication and Healthcare - Transforming Healthcare Through Innovations in Mobile and Wireless Technologies (MOBIHEALTH).

[34]  Yusheng Ji,et al.  RF-Sensing of Activities from Non-Cooperative Subjects in Device-Free Recognition Systems Using Ambient and Local Signals , 2014, IEEE Transactions on Mobile Computing.

[35]  Anja S. Göritz,et al.  Plain Texts as an Online Mood-Induction Procedure , 2009 .

[36]  Y. Lin,et al.  An Experimental Study on Physiological Parameters Toward Driver Emotion Recognition , 2007, HCI.

[37]  Jennifer Healey,et al.  SmartCar: detecting driver stress , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[38]  Kiraz Candan Herdem Reactions: Twitter based mobile application for awareness of friends' emotions , 2012, UbiComp.

[39]  Sasu Tarkoma,et al.  Accelerometer-based transportation mode detection on smartphones , 2013, SenSys '13.

[40]  Shin'ichi Konomi Colocation networks: exploring the use of social andgeographical patterns in context-aware services , 2011, UbiComp '11.

[41]  Björn W. Schuller,et al.  New Avenues in Opinion Mining and Sentiment Analysis , 2013, IEEE Intelligent Systems.

[42]  Fadel Adib,et al.  See through walls with WiFi! , 2013, SIGCOMM.

[43]  Emiliano Miluzzo,et al.  A survey of mobile phone sensing , 2010, IEEE Communications Magazine.

[44]  J. Beatty,et al.  The pupillary system. , 2000 .

[45]  T. Dalgleish,et al.  Handbook of cognition and emotion , 1999 .

[46]  C. Collet,et al.  Autonomic nervous system response patterns specificity to basic emotions. , 1997, Journal of the Autonomic Nervous System.

[47]  Robert Li Kam Wa MoodScope: Building a Mood Sensor from Smartphone Usage Patterns , 2012 .

[48]  Johannes Schöning,et al.  Informing intelligent user interfaces by inferring affective states from body postures in ubiquitous computing environments , 2013, IUI '13.

[49]  Shwetak N. Patel,et al.  Whole-home gesture recognition using wireless signals , 2013, MobiCom.

[50]  P. Wilhelm,et al.  Assessing Mood in Daily Life Structural Validity, Sensitivity to Change, and Reliability of a Short-Scale to Measure Three Basic Dimensions of Mood , 2007 .

[51]  C. Fisher,et al.  Using experience sampling methodology in organizational behavior. , 2012 .

[52]  Anja Bachmann,et al.  How to use smartphones for less obtrusive ambulatory mood assessment and mood recognition , 2015, UbiComp/ISWC Adjunct.

[53]  Jeffrey T. Hancock,et al.  Experimental evidence of massive-scale emotional contagion through social networks , 2014, Proceedings of the National Academy of Sciences.

[54]  Kai Zhao,et al.  CoSense: a collaborative sensing platform for mobile devices , 2013, SenSys '13.

[55]  Rosalind W Picard Recognizing Stress, Engagement, and Positive Emotion , 2015, IUI.

[56]  S. vanDongen Graph Clustering by Flow Simulation , 2000 .

[57]  S. Mourato,et al.  Happiness is greater in natural environments , 2013 .

[58]  John J. B. Allen,et al.  The handbook of emotion elicitation and assessment , 2007 .

[59]  Amit P. Sheth,et al.  Harnessing Twitter "Big Data" for Automatic Emotion Identification , 2012, 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing.

[60]  Jamie Payton,et al.  Recognizing social gestures with a wrist-worn smartband , 2015, 2015 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops).

[61]  Johan Koolwaaij,et al.  Identifying meaningful locations , 2006, 2006 Third Annual International Conference on Mobile and Ubiquitous Systems: Networking & Services.

[62]  Martin L. Griss,et al.  Activity-Aware Mental Stress Detection Using Physiological Sensors , 2010, MobiCASE.

[63]  Athanasios V. Vasilakos,et al.  Body Area Networks: A Survey , 2010, Mob. Networks Appl..

[64]  Wei Wang,et al.  Gait recognition using wifi signals , 2016, UbiComp.

[65]  Enzo Pasquale Scilingo,et al.  Recognizing Emotions Induced by Affective Sounds through Heart Rate Variability , 2015, IEEE Transactions on Affective Computing.

[66]  Eiman Kanjo,et al.  A supermarket stress map , 2013, UbiComp.

[67]  Georg Brügner,et al.  Emotions in everyday life: an ambulatory monitoring study with female students , 2005, Biological Psychology.

[68]  David Fitzpatrick,et al.  Physiological Changes Associated with Emotion , 2001 .

[69]  Dan Wu,et al.  WiDir: walking direction estimation using wireless signals , 2016, UbiComp.

[70]  Adam D. I. Kramer,et al.  Detecting Emotional Contagion in Massive Social Networks , 2014, PloS one.

[71]  Oscar Mayora-Ibarra,et al.  Smartphone-Based Recognition of States and State Changes in Bipolar Disorder Patients , 2015, IEEE Journal of Biomedical and Health Informatics.

[72]  Marc Schröder,et al.  Issues in emotion-oriented computing – towards a shared understanding , 2006 .

[73]  Katharine S. Willis,et al.  WiMo: location-based emotion tagging , 2009, MUM.

[74]  Vít Pászto,et al.  Mapping Emotions: Spatial Distribution of Safety Perception in the City of Olomouc , 2017 .

[75]  E. Scott Geller,et al.  Measuring road rage: development of the Propensity for Angry Driving Scale , 2001 .

[76]  Daniela Fogli,et al.  Affective geographies: toward a richer cartographic semantics for the geospatial web , 2008, AVI '08.

[77]  Michael E. Dawson,et al.  Startle Modification: Implications for Neuroscience, Cognitive Science, and Clinical Science , 2008 .

[78]  Romit Roy Choudhury,et al.  Your reactions suggest you liked the movie: automatic content rating via reaction sensing , 2013, UbiComp.

[79]  Olle Hilborn,et al.  A Serious Game using Physiological Interfaces for Emotion regulation Training in the Context of Financial Decision-Making , 2012, ECIS.

[80]  Mike Thelwall,et al.  Twitter, MySpace, Digg: Unsupervised Sentiment Analysis in Social Media , 2012, TIST.

[81]  Taghi M. Khoshgoftaar,et al.  Impact of Feature Selection Techniques for Tweet Sentiment Classification , 2015, FLAIRS.

[82]  Dan Wu,et al.  Human respiration detection with commodity wifi devices: do user location and body orientation matter? , 2016, UbiComp.

[83]  Mirco Musolesi,et al.  Trajectories of depression: unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis , 2015, UbiComp.

[84]  Björn W. Schuller,et al.  Emotion on the Road - Necessity, Acceptance, and Feasibility of Affective Computing in the Car , 2010, Adv. Hum. Comput. Interact..