Analysis and Use of the Emotional Context with Wearable Devices for Games and Intelligent Assistants

In this paper, we consider the use of wearable sensors for providing affect-based adaptation in Ambient Intelligence (AmI) systems. We begin with discussion of selected issues regarding the applications of affective computing techniques. We describe our experiments for affect change detection with a range of wearable devices, such as wristbands and the BITalino platform, and discuss an original software solution, which we developed for this purpose. Furthermore, as a test-bed application for our work, we selected computer games. We discuss the state-of-the-art in affect-based adaptation in games, described in terms of the so-called affective loop. We present our original proposal of a conceptual design framework for games, called the affective game design patterns. As a proof-of-concept realization of this approach, we discuss some original game prototypes, which we have developed, involving emotion-based control and adaptation. Finally, we comment on a software framework, that we have previously developed, for context-aware systems which uses human emotional contexts. This framework provides means for implementing adaptive systems using mobile devices with wearable sensors.

[1]  Andy Adler,et al.  Validation of the Empatica E4 wristband , 2016, 2016 IEEE EMBS International Student Conference (ISC).

[2]  Grzegorz J. Nalepa,et al.  Uncertain context data management in dynamic mobile environments , 2017, Future Gener. Comput. Syst..

[3]  Nico Pallamin,et al.  Emotion Recognition Using Physiological Signals: Laboratory vs. Wearable Sensors , 2017, AHFE.

[4]  Yoshiro Tajitsu,et al.  Piezoelectret sensor made from an electro-spun fluoropolymer and its use in a wristband for detecting heart-beat signals , 2015, IEEE Transactions on Dielectrics and Electrical Insulation.

[5]  Vicente Julián,et al.  Cognitive assistants , 2018, Int. J. Hum. Comput. Stud..

[6]  Senem Velipasalar,et al.  A More Complete Picture of Emotion Using Electrocardiogram and Electrodermal Activity to Complement Cognitive Data , 2016, HCI.

[7]  M. Bartlett,et al.  Machine Analysis of Facial Expressions , 2007 .

[8]  A. Grabowska,et al.  The Nencki Affective Picture System (NAPS): Introduction to a novel, standardized, wide-range, high-quality, realistic picture database , 2013, Behavior research methods.

[9]  Mohsen Nabian,et al.  Facial Expression and Peripheral Physiology Fusion to Decode Individualized Affective Experience , 2018, AffComp@IJCAI.

[10]  Likun Xia,et al.  Association of mental stress with video games , 2012, 2012 4th International Conference on Intelligent and Advanced Systems (ICIAS2012).

[11]  Andrea Bonarini,et al.  Modeling enjoyment preference from physiological responses in a car racing game , 2010, Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games.

[12]  J. Gratch,et al.  The Oxford Handbook of Affective Computing , 2014 .

[13]  Carlos Carrascosa,et al.  Intelligent Wristbands for the Automatic Detection of Emotional States for the Elderly , 2018, IDEAL.

[14]  Jean-Philippe Thiran,et al.  Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data , 2015, Pattern Recognit. Lett..

[15]  E. Hudlicka AFFECTIVE COMPUTING FOR GAME DESIGN , 2008 .

[16]  Peter Goldie,et al.  Emotions, feelings and intentionality , 2002 .

[17]  J. Russell A circumplex model of affect. , 1980 .

[18]  Juha Röning,et al.  Early detection of migraine attacks based on wearable sensors: experiences of data collection using Empatica E4 , 2017, UbiComp/ISWC Adjunct.

[19]  P. Ekman Facial expression and emotion. , 1993, The American psychologist.

[20]  Rosalind W. Picard,et al.  Non-contact, automated cardiac pulse measurements using video imaging and blind source separation , 2022 .

[21]  Grzegorz J. Nalepa,et al.  HEARTDROID—Rule engine for mobile and context‐aware expert systems , 2018, Expert Syst. J. Knowl. Eng..

[22]  Grzegorz J. Nalepa,et al.  Towards the Development of Sensor Platform for Processing Physiological Data from Wearable Sensors , 2018, ICAISC.

[23]  Carlos Carrascosa,et al.  Detecting emotions through non-invasive wearables , 2018, Log. J. IGPL.

[24]  Fabio Babiloni,et al.  How to Measure Cerebral Correlates of Emotions in Marketing Relevant Tasks , 2014, Cognitive Computation.

[25]  Grzegorz J. Nalepa,et al.  Affective Computing Experiments in Virtual Reality with Wearable Sensors. Methodological considerations and preliminary results , 2016, AfCAI.

[26]  Chuan-Yu Chang,et al.  Emotion recognition with consideration of facial expression and physiological signals , 2009, 2009 IEEE Symposium on Computational Intelligence in Bioinformatics and Computational Biology.

[27]  J. Prinz Gut Reactions: A Perceptual Theory of Emotion , 2004 .

[28]  Grzegorz J. Nalepa,et al.  Incomplete and Uncertain Data Handling in Context-Aware Rule-Based Systems with Modified Certainty Factors Algebra , 2014, RuleML.

[29]  Ricardo Gutierrez-Osuna,et al.  Physiological Modalities for Relaxation Skill Transfer in Biofeedback Games , 2017, IEEE Journal of Biomedical and Health Informatics.

[30]  Rosalind W. Picard,et al.  Expression glasses: a wearable device for facial expression recognition , 1999, CHI Extended Abstracts.

[31]  Thierry Pun,et al.  Multimodal Emotion Recognition in Response to Videos , 2012, IEEE Transactions on Affective Computing.

[32]  Mohammad A. Khalilzadeh,et al.  EMOTIONAL STRESS RECOGNITION SYSTEM FOR AFFECTIVE COMPUTING BASED ON BIO-SIGNALS , 2010 .

[33]  E. Brown,et al.  Psychophysiological signals associated with affective states , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[34]  Sylvia D. Kreibig,et al.  Autonomic nervous system activity in emotion: A review , 2010, Biological Psychology.

[35]  Wanqing Li,et al.  A Real-Time Facial Expression Recognition System for Online Games , 2008, Int. J. Comput. Games Technol..

[36]  Eva Hudlicka,et al.  Affective game engines: motivation and requirements , 2009, FDG.

[37]  J. Prinz Which Emotions Are Basic , 2007 .

[38]  Fabien Ringeval,et al.  AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge , 2016, AVEC@ACM Multimedia.

[39]  Grzegorz J. Nalepa,et al.  BandReader - A Mobile Application for Data Acquisition from Wearable Devices in Affective Computing Experiments , 2018, 2018 11th International Conference on Human System Interaction (HSI).

[40]  Gregory D. Abowd,et al.  Providing architectural support for building context-aware applications , 2000 .

[41]  Davide Fossati,et al.  Affect detection from non-stationary physiological data using ensemble classifiers , 2014, Evolving Systems.

[42]  M. Bradley,et al.  Measuring emotion: Behavior, feeling, and physiology , 2000 .

[43]  Lennart E. Nacke,et al.  Affective Ludology, Flow and Immersion in a First- Person Shooter: Measurement of Player Experience , 2009, ArXiv.

[44]  Regan L. Mandryk,et al.  A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies , 2007, Int. J. Hum. Comput. Stud..

[45]  David Capelo Chaves Caminha Development of Emotional Game Mechanics through the use of Biometric Sensors , 2017 .

[46]  Grzegorz J. Nalepa,et al.  Mobile platform for affective context-aware systems , 2019, Future Gener. Comput. Syst..

[47]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[48]  Grzegorz J. Nalepa,et al.  Prototypes of Arcade Games Enabling Affective Interaction , 2019, ICAISC.

[49]  Paul Griffiths,et al.  Emotions in the wild: The situated perspective on emotion , 2005 .

[50]  Jason Williams,et al.  Emotion Recognition Using Bio-sensors: First Steps towards an Automatic System , 2004, ADS.

[51]  M. Cabanac What is emotion? , 2002, Behavioural Processes.

[52]  George Trigeorgis,et al.  End-to-End Multimodal Emotion Recognition Using Deep Neural Networks , 2017, IEEE Journal of Selected Topics in Signal Processing.

[53]  Yuri Ivanov,et al.  Probabilistic combination of multiple modalities to detect interest , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[54]  Zhilin Zhang,et al.  Photoplethysmography-Based Heart Rate Monitoring in Physical Activities via Joint Sparse Spectrum Reconstruction , 2015, IEEE Transactions on Biomedical Engineering.

[55]  Chao Xu,et al.  Emotion Recognition Based on Multichannel Physiological Signals with Comprehensive Nonlinear Processing , 2018, Sensors.

[56]  Inho Choi,et al.  Multimodal data collection framework for mental stress monitoring , 2017, UbiComp/ISWC Adjunct.

[57]  Lennart E. Nacke,et al.  Brain Training for Silver Gamers: Effects of Age and Game Form on Effectiveness, Efficiency, Self-Assessment, and Gameplay Experience , 2009, Cyberpsychology Behav. Soc. Netw..

[58]  Grzegorz J. Nalepa,et al.  Uncertainty handling in rule-based mobile context-aware systems , 2017, Pervasive Mob. Comput..

[59]  Rosalind W. Picard,et al.  Evaluating affective interactions: Alternatives to asking what users feel , 2005 .

[60]  Isabela Granic,et al.  Designing and Utilizing Biofeedback Games for Emotion Regulation: The Case of Nevermind , 2016, CHI Extended Abstracts.

[61]  Grzegorz J. Nalepa,et al.  Affective Design Patterns in Computer Games. Scrollrunner Case Study , 2017, FedCSIS.

[62]  Alan J. Dix,et al.  Affective Videogames and Modes of Affective Gaming: Assist Me, Challenge Me, Emote Me (ACE) , 2005, DiGRA Conference.

[63]  Ioanna Chouvarda,et al.  A Computer-Assisted System with Kinect Sensors and Wristband Heart Rate Monitors for Group Classes of Exercise-Based Rehabilitation , 2018 .

[64]  J. Singer,et al.  Cognitive, social, and physiological determinants of emotional state. , 1962, Psychological review.

[65]  C. Izard Basic emotions, relations among emotions, and emotion-cognition relations. , 1992, Psychological review.

[66]  Regan Lee Mandryk Evaluating Affective Computing Environments Using Physiological Measures , 2005 .

[67]  Matjaz Gams,et al.  Monitoring stress with a wrist device using context , 2017, J. Biomed. Informatics.

[68]  William M. Campbell,et al.  Multi-Modal Audio, Video and Physiological Sensor Learning for Continuous Emotion Prediction , 2016, AVEC@ACM Multimedia.

[69]  M. Mahadevappa,et al.  Emotion recognition based on physiological signals using valence-arousal model , 2015, 2015 Third International Conference on Image Information Processing (ICIIP).

[70]  Boyan Bontchev,et al.  Adaptation in Affective Video Games: A Literature Review , 2016 .

[71]  Georgios N. Yannakakis,et al.  Multimodal PTSD characterization via the StartleMart game , 2014, Journal on Multimodal User Interfaces.

[72]  Shao-Kuo Tai,et al.  Emotion stress detection using EEG signal and deep learning technologies , 2018, 2018 IEEE International Conference on Applied System Invention (ICASI).

[73]  Regan L. Mandryk,et al.  Using affective state to adapt characters, NPCs, and the environment in a first-person shooter game , 2014, 2014 IEEE Games Media Entertainment.

[74]  Anind K. Dey,et al.  Designing mediation for context-aware applications , 2005, TCHI.

[75]  W. James II.—WHAT IS AN EMOTION ? , 1884 .

[76]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[77]  R. Lazarus From psychological stress to the emotions: a history of changing outlooks. , 1993, Annual review of psychology.

[78]  Guillaume Chanel,et al.  A review of the use of psychophysiological methods in game research , 2011 .

[79]  Mohanasankar Sivaprakasam,et al.  Design, development and clinical validation of a wrist-based optical heart rate monitor , 2016, 2016 IEEE International Symposium on Medical Measurements and Applications (MeMeA).

[80]  Nicola Guarino,et al.  Formal Ontology and Information Systems , 1998 .

[81]  José Vasconcelos-Raposo,et al.  Heart Rate, Electrodermic Activity and Self-Reported Experience in Shooter Games, in University Students , 2018, PSYCHTECH & HEALTH JOURNAL.

[82]  Fabien Ringeval,et al.  Summary for AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge , 2016, ACM Multimedia.