A User-Modeling Approach to Build User's Psycho-Physiological Maps of Emotions using Bio-Sensors

Near to real-time emotion recognition is a promising task for human-computer interaction (HCI) and human-robot interaction (HRI). Using knowledge about the user's emotions depends upon the possibility to extract information about users' emotions during HCI or HRI without explicitly asking users about the feelings they are experiencing. To be able to sense the user's emotions without interrupting the HCI, we present a new method applied to the emotional experience of the user for extracting semantic information from the autonomic nervous system (ANS) signals associated with emotions. We use the concepts of 1st person - where the subject consciously (and subjectively) extracts the semantic meaning of a given lived experience, (e.g. `I felt amused') - and 3rd person approach - where the experimenter interprets the semantic meaning of the subject's experience from a set of externally (and objectively) measured variables (e.g. galvanic skin response measures). Based on the 3rd person approach, our technique aims at psychologically interpreting physiological parameters (skin conductance and heart rate), and at producing a continuous extraction of the user's affective state during HCI or HRI. We also combine it with the 1st person approach measure which allows a tailored interpretation of the physiological measure closely related to the user own emotional experience

[1]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  A. Ohman,et al.  "Unconscious anxiety": phobic responses to masked stimuli. , 1994, Journal of abnormal psychology.

[3]  Garmt Dijksterhuis,et al.  Differential human electrodermal responses to odours , 2003, Neuroscience Letters.

[4]  M. Bradley,et al.  Brain potentials in affective picture processing: covariation with autonomic arousal and affective report , 2000, Biological Psychology.

[5]  Ieee Xplore,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence Information for Authors , 2022, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  R. Simons,et al.  Emotional imagery and physical anhedonia. , 1994, Psychophysiology.

[7]  Adson Ferreira da Rocha,et al.  A tool for time-frequency analysis of heart rate variability , 2003, Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE Cat. No.03CH37439).

[8]  H. Critchley.  Review: Electrodermal Responses: What Happens in the Brain , 2002 .

[9]  H. Critchley.  Electrodermal responses: what happens in the brain. , 2002, The Neuroscientist : a review journal bringing neurobiology, neurology and psychiatry.

[10]  Mitsuru Kikuchi,et al.  Changes in EEG and autonomic nervous activity during meditation and their association with personality traits. , 2005, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[11]  C. Rennie,et al.  Decomposing skin conductance into tonic and phasic components. , 1997, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[12]  Joni Kettunen,et al.  METHODOLOGICAL AND EMPIRICAL ADVANCES IN THE QUANTITATIVE ANALYSIS OF SPONTANEOUS RESPONSES IN PSYCHOPHYSIOLOGICAL TIME SERIES , 1999 .

[13]  Changchun Liu,et al.  An empirical study of machine learning techniques for affect recognition in human–robot interaction , 2006, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[14]  Gari D. Clifford,et al.  Signal processing methods for heart rate variability , 2002 .

[15]  S. Khalfa,et al.  Event-related skin conductance responses to musical emotions in humans , 2002, Neuroscience Letters.

[16]  Jaakko Malmivuo,et al.  The Electrodermal Response , 1995 .

[17]  Bruce H. Friedman,et al.  Multivariate Discrimination of Emotion-Specific Autonomic Nervous System Activity , 2002 .

[18]  Robert Plonsey,et al.  Bioelectromagnetism: Principles and Applications of Bioelectric and Biomagnetic Fields , 1995 .

[19]  L Keltikangas-Järvinen,et al.  Intraindividual analysis of instantaneous heart rate variability. , 2001, Psychophysiology.

[20]  J. Russell.  A circumplex model of affect. , 1980 .

[21]  Olivier Meste,et al.  Time-varying analysis methods and models for the respiratory and cardiac system coupling in graded exercise , 2005, IEEE Transactions on Biomedical Engineering.

[22]  K. H. Kim,et al.  Emotion recognition system using short-term monitoring of physiological signals , 2004, Medical and Biological Engineering and Computing.

[23]  Christine L. Lisetti,et al.  Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals , 2004, EURASIP J. Adv. Signal Process..

[24]  M. Bradley,et al.  Looking at pictures: affective, facial, visceral, and behavioral reactions. , 1993, Psychophysiology.

[25]  Olivier Villon,et al.  Toward Building Adaptive User's Psycho-Physiological Maps of Emotions using Bio-Sensors , 2006 .

[26]  A. Angrilli,et al.  Cardiac responses associated with affective processing of unpleasant film stimuli. , 2000, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[27]  G. Breithardt,et al.  Heart rate variability: standards of measurement, physiological interpretation and clinical use. Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology. , 1996 .

[28]  A. Malliani,et al.  Heart rate variability. Standards of measurement, physiological interpretation, and clinical use , 1996 .

[29]  Catherine M. Stoney,et al.  Depressed Mood Is Related to High-Frequency Heart Rate Variability During Stressors , 2000, Psychosomatic medicine.

[30]  I. Christie,et al.  Autonomic specificity of discrete emotion and dimensions of affective space: a multivariate approach. , 2004, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[31]  J. Cacioppo,et al.  Inferring psychological significance from physiological signals. , 1990, The American psychologist.

[32]  Changchun Liu,et al.  An empirical study of machine learning techniques for affect recognition in human-robot interaction , 2005 .

[33]  Peter Jönsson,et al.  The effects of pictures of emotional faces on tonic and phasic autonomic cardiac control in women and men , 2003, Biological Psychology.

[34]  J F Thayer,et al.  Autonomic balance revisited: panic anxiety and heart rate variability. , 1998, Journal of psychosomatic research.

[35]  P. Venables,et al.  Publication recommendations for electrodermal measurements. , 1981 .

[36]  M. Bradley,et al.  Affective reactions to acoustic stimuli. , 2000, Psychophysiology.

[37]  R. Mccraty,et al.  The effects of emotions on short-term power spectrum analysis of heart rate variability . , 1995, The American journal of cardiology.

[38]  Nilanjan Sarkar,et al.  Affect-sensitive human-robot cooperation - theory and experiments , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[39]  P. Venables,et al.  Committee report. Publication recommendations for electrodermal measurements. , 1981, Psychophysiology.

[40]  M. Bradley,et al.  Neuroanatomical correlates of pleasant and unpleasant emotion , 1997, Neuropsychologia.