DeepFlow: Detecting Optimal User Experience From Physiological Data Using Deep Neural Networks

The affective state called flow is described as a state of optimal experience, total immersion and high productivity. As an important metric for various scenarios ranging from (professional) sports to work environments to user experience evaluations, it is extensively studied using traditional questionnaires. In order to make flow measurement accessible for online, real-time environments, in this work, we present our preliminary findings towards automatically estimating a user's flow state based on physiological signals measured with a wearable device. We conducted a study of subjects playing the game Tetris in varying difficulty levels, leading to boredom, stress, and flow. Using a convolutional neural network, we achieve an accuracy of 70% in recognizing flow-inducing levels. In the future, we expect flow to be a potential reward signal for human-in-the-loop reinforcement learning systems.

[1]  C. Kozyrakis,et al.  TETRIS , 2017 .

[2]  Enzo Pasquale Scilingo,et al.  Wearable Monitoring for Mood Recognition in Bipolar Disorder Based on History-Dependent Long-Term Heart Rate Variability Analysis , 2014, IEEE Journal of Biomedical and Health Informatics.

[3]  Rosalind W. Picard Affective Computing for HCI , 1999, HCI.

[4]  A. Flatt,et al.  Ultra-short-term heart rate variability indexes at rest and post-exercise in athletes: evaluating the agreement with accepted recommendations. , 2014, Journal of sports science & medicine.

[5]  D. Eichel Beyond Boredom And Anxiety Experiencing Flow In Work And Play , 2016 .

[6]  H. Marsh,et al.  Development and Validation of a Scale to Measure Optimal Experience: The Flow State Scale , 1996 .

[7]  Douglas Eck,et al.  Learning via social awareness: improving sketch representations with facial feedback , 2018, ICLR.

[8]  Andy Adler,et al.  Validation of the Empatica E4 wristband , 2016, 2016 IEEE EMBS International Student Conference (ISC).

[9]  A. Barreto,et al.  Stress Detection in Computer Users Based on Digital Signal Processing of Noninvasive Physiological Variables , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[10]  K. W. Kallus,et al.  Human Factors Analysis Using Wearable Sensors in the Context of Cognitive and Emotional Arousal , 2015 .

[11]  Juha Röning,et al.  Early detection of migraine attacks based on wearable sensors: experiences of data collection using Empatica E4 , 2017, UbiComp/ISWC Adjunct.

[12]  J. Matias Kivikangas,et al.  Psychophysiology of flow experience : An explorative study , 2006 .

[13]  Lennart E. Nacke,et al.  Flow and immersion in first-person shooters: measuring the player's gameplay experience , 2008, Future Play.

[14]  M. Erb,et al.  Brain activity underlying emotional valence and arousal: A response‐related fMRI study , 2004, Human brain mapping.

[15]  Mykola Pechenizkiy,et al.  Stress detection from speech and Galvanic Skin Response signals , 2013, Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems.

[16]  Guillaume Chanel,et al.  Emotion Assessment From Physiological Signals for Adaptation of Game Difficulty , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[17]  George V. N. Dearborn Review of General psychology. , .

[18]  S. Globali,et al.  IEEE INTELLIGENT SYSTEMS , 2022, IEEE MultiMedia.

[19]  Min Xu,et al.  Affective content analysis in comedy and horror videos by audio emotional event detection , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[20]  Alexander Maedche,et al.  Got Flow?: Using Machine Learning on Physiological Data to Classify Flow , 2018, CHI Extended Abstracts.

[21]  Geoffrey H. Tison,et al.  DeepHeart: Semi-Supervised Sequence Learning for Cardiovascular Risk Prediction , 2018, AAAI.

[22]  Emmanuel Dellandréa,et al.  LIRIS-ACCEDE: A Video Database for Affective Content Analysis , 2015, IEEE Transactions on Affective Computing.

[23]  Qiang Ji,et al.  Video Affective Content Analysis: A Survey of State-of-the-Art Methods , 2015, IEEE Transactions on Affective Computing.

[24]  Kewei Chen,et al.  Neural correlates of heart rate variability during emotion , 2009, NeuroImage.

[25]  D. Hoang FLOW: The Psychology of Optimal Experience , 2018 .

[26]  F. Ullén,et al.  Physiological correlates of the flow experience during computer game playing. , 2015, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[27]  Dieter Kleinböhl,et al.  Physiological aspects of flow experiences: Skills-demand-compatibility effects on heart rate variability and salivary cortisol☆ , 2011 .

[28]  IEEE conference on computer vision and pattern recognition , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[29]  Mohammad H. Mahoor,et al.  Facial Expression Recognition from World Wild Web , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[30]  Sylvie Charbonnier,et al.  A comparison of wearable and stationary sensors for stress detection , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[31]  Mohammad H. Mahoor,et al.  AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild , 2017, IEEE Transactions on Affective Computing.

[32]  Guillaume Chanel,et al.  Boredom, engagement and anxiety as indicators for adaptation to difficulty in games , 2008, MindTrek '08.

[33]  Rosalind W. Picard Affective computing: challenges , 2003, Int. J. Hum. Comput. Stud..

[34]  F. Shaffer,et al.  An Overview of Heart Rate Variability Metrics and Norms , 2017, Front. Public Health.

[35]  Christine L. Lisetti Affective computing , 1998, Pattern Analysis and Applications.

[36]  Luca Citi,et al.  Characterization of Depressive States in Bipolar Patients Using Wearable Textile Technology and Instantaneous Heart Rate Variability Assessment , 2015, IEEE Journal of Biomedical and Health Informatics.

[37]  Vladlen Koltun,et al.  An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling , 2018, ArXiv.

[38]  Nico Pallamin,et al.  Emotion Recognition Using Physiological Signals: Laboratory vs. Wearable Sensors , 2017, AHFE.

[39]  Luca Citi,et al.  Revealing Real-Time Emotional Responses: a Personalized Assessment based on Heartbeat Dynamics , 2014, Scientific Reports.

[40]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .