Mobile platform for affective context-aware systems

Abstract In our work, we focus on detection of affective states, their proper identification and interpretation with use of wearable and mobile devices. We propose a data acquisition layer based on wearable devices able to gather physiological data, and we integrate it with mobile context-aware framework. Furthermore, we formulate a method for personalization of emotion detection. This solution offers a non-intrusive measurement thanks to the use of wearable devices, such as wristbands. As means of validation of our concepts we describe a series of experiments that we conducted.

[1]  Daniel McDuff,et al.  A survey of remote optical photoplethysmographic imaging methods , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[2]  Gio Wiederhold,et al.  Mediators in the architecture of future information systems , 1992, Computer.

[3]  Grzegorz J. Nalepa,et al.  Uncertainty handling in rule-based mobile context-aware systems , 2017, Pervasive Mob. Comput..

[4]  W. James,et al.  What Is an Emotion , 1977 .

[5]  Gregory D. Abowd,et al.  Providing architectural support for building context-aware applications , 2000 .

[6]  Jesse Hoey,et al.  An Exploratory Study into the Use of an Emotionally Aware Cognitive Assistant , 2015, AAAI Workshop: Artificial Intelligence Applied to Assistive Technologies and Smart Environments.

[7]  Anind K. Dey,et al.  Designing mediation for context-aware applications , 2005, TCHI.

[8]  Rafael Ramírez,et al.  Detecting Emotion from EEG Signals Using the Emotive Epoc Device , 2012, Brain Informatics.

[9]  Rosalind W Picard.  Recognizing Stress, Engagement, and Positive Emotion , 2015, IUI.

[10]  Bernd Kleinjohann,et al.  Real-Time Facial Expression Recognition Using a Fuzzy Emotion Model , 2007, 2007 IEEE International Fuzzy Systems Conference.

[11]  Byung-Soo Kim,et al.  A Design Platform for Emotion-Aware User Interfaces , 2014, ERM4HCI '14.

[12]  Franck Berthelon,et al.  Emotion ontology for context awareness , 2013, 2013 IEEE 4th International Conference on Cognitive Infocommunications (CogInfoCom).

[13]  Rosalind W. Picard.  Affective Computing , 1997 .

[14]  A. Muaremi,et al.  Towards Measuring Stress with Smartphones and Wearable Devices During Workday and Sleep , 2013, BioNanoScience.

[15]  A. Colosimo,et al.  EEG Frontal Asymmetry Related to Pleasantness of Olfactory Stimuli in Young Subjects , 2016 .

[16]  Stacy Marsella,et al.  EMA: A process model of appraisal dynamics , 2009, Cognitive Systems Research.

[17]  Grzegorz J. Nalepa,et al.  Uncertain context data management in dynamic mobile environments , 2017, Future Gener. Comput. Syst..

[18]  Thang Huynh Quyet,et al.  A real-time model based Support Vector Machine for emotion recognition through EEG , 2012, ICCA 2012.

[19]  Grzegorz J. Nalepa,et al.  Rule-based solution for context-aware reasoning on mobile devices , 2014, Comput. Sci. Inf. Syst..

[20]  Xingda Qu,et al.  Emotion Prediction from Physiological Signals: A Comparison Study Between Visual and Auditory Elicitors , 2014, Interact. Comput..

[21]  Maria Ebling,et al.  Can Cognitive Assistants Disappear? , 2016, IEEE Pervasive Comput..

[22]  Steve DiPaola,et al.  Exploring Different Ways of Navigating Emotionally Responsive Artwork in Immersive Virtual Environments , 2015, EVA.

[23]  R. Lazarus.  Psychological stress and the coping process , 1970 .

[24]  Atsuhiro Takasu,et al.  Bayesian probabilistic model for context-aware recommendations , 2015, iiWAS.

[25]  J. Prinz.  Gut Reactions: A Perceptual Theory of Emotion , 2004 .

[26]  John Hunt.  Guide to the Unified Process featuring UML, Java and Design Patterns , 2003, Springer Professional Computing.

[27]  Sung-Bae Cho,et al.  Bayesian Network-Based High-Level Context Recognition for Mobile Context Sharing in Cyber-Physical System , 2011, Int. J. Distributed Sens. Networks.

[28]  Joost Broekens,et al.  AffectButton: A method for reliable and valid affective self-report , 2013, Int. J. Hum. Comput. Stud..

[29]  J. Prinz.  Which Emotions Are Basic , 2007 .

[30]  Grzegorz J. Nalepa,et al.  Affective Design Patterns in Computer Games. Scrollrunner Case Study , 2017, FedCSIS.

[31]  B. Kröse,et al.  Bayesian Activity Recognition in Residence for Elders , 2007 .

[32]  Benoit Huet,et al.  Towards multimodal emotion recognition: a new approach , 2010, CIVR '10.

[33]  David Garlan,et al.  Context is key , 2005, CACM.

[34]  Winslow Burleson,et al.  ABE: An Agent-Based Software Architecture for a Multimodal Emotion Recognition Framework , 2011, 2011 Ninth Working IEEE/IFIP Conference on Software Architecture.

[35]  Gheorghe Tecuci,et al.  Toward cognitive assistants for complex decision making under uncertainty , 2014, Intell. Decis. Technol..

[36]  Christine Julien,et al.  Resource-Optimized Quality-Assured Ambiguous Context Mediation Framework in Pervasive Environments , 2012, IEEE Transactions on Mobile Computing.

[37]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[38]  Christine L. Lisetti,et al.  Toward multimodal fusion of affective cues , 2006, HCM '06.

[39]  J. Cacioppo,et al.  The psychophysiology of emotion. , 1993 .

[40]  Sylvia D. Kreibig,et al.  Autonomic nervous system activity in emotion: A review , 2010, Biological Psychology.

[41]  Fabien Ringeval,et al.  AV+EC 2015: The First Affect Recognition Challenge Bridging Across Audio, Video, and Physiological Data , 2015, AVEC@ACM Multimedia.

[42]  P. Young,et al.  Emotion and personality , 1963 .

[43]  Michael D. Robinson,et al.  Measures of emotion: A review , 2009, Cognition & emotion.

[44]  A. Grabowska,et al.  The Nencki Affective Picture System (NAPS): Introduction to a novel, standardized, wide-range, high-quality, realistic picture database , 2013, Behavior research methods.

[45]  Jennifer Healey.  Physiological Sensing of Emotion , 2015 .

[46]  Akane Sano,et al.  Stress Recognition Using Wearable Sensors and Mobile Phones , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[47]  Zakaria Maamar,et al.  A context-based mediation approach to compose semantic Web services , 2007, TOIT.

[48]  Katarzyna Wac,et al.  Ambulatory Assessment of Affect: Survey of Sensor Systems for Monitoring of Autonomic Nervous Systems Activation in Emotion , 2014, IEEE Transactions on Affective Computing.

[49]  J. Russell,et al.  An approach to environmental psychology , 1974 .

[50]  Hung Keng Pung,et al.  A middleware for building context-aware mobile services , 2004, 2004 IEEE 59th Vehicular Technology Conference. VTC 2004-Spring (IEEE Cat. No.04CH37514).

[51]  Grzegorz J. Nalepa,et al.  AfCAI systems: Affective Computing with Context Awareness for Ambient Intelligence. Research proposal , 2016, AfCAI.

[52]  Mariano Alcañiz Raya,et al.  Assessing brain activations associated with emotional regulation during virtual reality mood induction procedures , 2015, Expert Syst. Appl..

[53]  Carlos Carrascosa,et al.  Emotions Detection on an Ambient Intelligent System Using Wearable Devices , 2019, AfCAI.

[54]  Fabio Babiloni,et al.  How to Measure Cerebral Correlates of Emotions in Marketing Relevant Tasks , 2014, Cognitive Computation.

[55]  Tzyy-Ping Jung,et al.  The Wearable Multimodal Monitoring System: A Platform to Study Falls and Near-Falls in the Real-World , 2015, HCI.

[56]  Carlos Carrascosa,et al.  Using Non-invasive Wearables for Detecting Emotions with Intelligent Agents , 2016, SOCO-CISIS-ICEUTE.

[57]  Konstantinos N. Plataniotis,et al.  Affect recognition using EEG signal , 2012, 2012 IEEE 14th International Workshop on Multimedia Signal Processing (MMSP).

[58]  J. Al-Muhtadi,et al.  Mobile Gaia: a middleware for ad-hoc pervasive computing , 2005, Second IEEE Consumer Communications and Networking Conference, 2005. CCNC. 2005.

[59]  Eyke Hüllermeier,et al.  Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty , 2014, Inf. Sci..

[60]  Ayman El-Baz,et al.  Using physiological signal analysis to design affective VR games , 2015, 2015 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT).

[61]  Bill N. Schilit,et al.  Context-aware computing applications , 1994, Workshop on Mobile Computing Systems and Applications.

[62]  J. Russell.  A circumplex model of affect. , 1980 .

[63]  Qiang Ji,et al.  Active affective State detection and user assistance with dynamic bayesian networks , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[64]  Szymon Bobek,et al.  Framework for Benchmarking Rule-Based Inference Engines , 2017, ICAISC.

[65]  Henry A. Kautz,et al.  Learning and inferring transportation routines , 2004, Artif. Intell..

[66]  Denzil Ferreira.  AWARE:a mobile context instrumentation middleware to collaboratively understand human behavior , 2013 .

[67]  Gheorghe Tecuci,et al.  Knowledge Engineering: Building Cognitive Assistants for Evidence-based Reasoning , 2016 .

[68]  Kurt Geihs,et al.  Playing MUSIC — building context‐aware and self‐adaptive mobile applications , 2013, Softw. Pract. Exp..

[69]  Ipke Wachsmuth,et al.  Affective computing with primary and secondary emotions in a virtual human , 2009, Autonomous Agents and Multi-Agent Systems.

[70]  Wen Yao,et al.  Leveraging complex event processing for smart hospitals using RFID , 2011, J. Netw. Comput. Appl..

[71]  Thomas Fritz,et al.  Stuck and Frustrated or in Flow and Happy: Sensing Developers' Emotions and Progress , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[72]  Harry Chen,et al.  Semantic Web in the context broker architecture , 2004, Second IEEE Annual Conference on Pervasive Computing and Communications, 2004. Proceedings of the.

[73]  Svetha Venkatesh,et al.  Tracking and Surveillance in Wide-Area Spatial Environments Using the Abstract Hidden Markov Model , 2001, Int. J. Pattern Recognit. Artif. Intell..

[74]  Daniel McDuff,et al.  BioInsights: Extracting personal data from “Still” wearable motion sensors , 2015, 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[75]  Roy H. Campbell,et al.  Use of Ontologies in Pervasive Computing Environments , 2003 .

[76]  Iakovos S. Venieris,et al.  Context-aware service engineering: A survey , 2009, J. Syst. Softw..

[77]  K. Scherer.  What are emotions? And how can they be measured? , 2005 .

[78]  Stacy Marsella,et al.  A domain-independent framework for modeling emotion , 2004, Cognitive Systems Research.

[79]  Anind K. Dey.  Modeling and intelligibility in ambient environments , 2009, J. Ambient Intell. Smart Environ..

[80]  Yongzhao Zhan,et al.  Using Kinect for real-time emotion recognition via facial expressions , 2015, Frontiers of Information Technology & Electronic Engineering.