Affect detection from non-stationary physiological data using ensemble classifiers

Affect detection from physiological signals has received considerable attention. One challenge is that physiological measures exhibit considerable variations over time, making classification of future data difficult. The present study addresses this issue by providing insights on how diagnostic physiological features of affect change over time. Affective physiological data (electrocardiogram, electromyogram, skin conductivity, and respiration) was collected from four participants over five sessions each. Classification performance of a number of training strategies, under different conditions of features selection and engineering, were compared using an adaptive classifier ensemble algorithm. Analysis of the performance of individual physiological channels for affect detection is also provided. The key result is that using pooled features set for affect detection is more accurate than using day-specific features. A decision fusion strategy which combines decisions from classifiers trained on individual channels data outperformed a features fusion strategy. Results also show that the performance of the ensemble is affected by the choice of the base classifier and the alpha factor used to update the member classifiers of the ensemble. Finally, the corrugator and zygomatic facial EMGs were found to be more reliable measures for detecting the valence component of affect compared to other channels.

[1]  Stephen H. Fairclough,et al.  A research agenda for physiological computing , 2004, Interact. Comput..

[2]  Pramod K. Varshney,et al.  Modern Approaches in Applied Intelligence - 24th International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2011, Syracuse, NY, USA, June 28 - July 1, 2011, Proceedings, Part I , 2011, IEA/AIE.

[3]  G. A. Mendelsohn,et al.  Affect grid : A single-item scale of pleasure and arousal , 1989 .

[4]  Philippe Georges Zimmermann,et al.  Valence lasts longer than arousal : persistence of induced moods as assessed by psychophysiological measures , 2009 .

[5]  E. Vyzas,et al.  Affective Pattern Classification , 2002 .

[6]  Gregory Ditzler,et al.  Learning in Nonstationary Environments: A Survey , 2015, IEEE Computational Intelligence Magazine.

[7]  Rafael A. Calvo,et al.  Affective Modeling from Multichannel Physiology: Analysis of Day Differences , 2011, ACII.

[8]  Laila Dybkjær,et al.  Affective Dialogue Systems , 2004, Lecture Notes in Computer Science.

[9]  David A. Cieslak,et al.  A framework for monitoring classifiers’ performance: when and why failure occurs? , 2009, Knowledge and Information Systems.

[10]  Anil K. Jain,et al.  Statistical Pattern Recognition: A Review , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  T. Subba Rao,et al.  Classification, Parameter Estimation and State Estimation: An Engineering Approach Using MATLAB , 2004 .

[13]  P. Lang International affective picture system (IAPS) : affective ratings of pictures and instruction manual , 2005 .

[14]  Mincheol Whang,et al.  A Physiological Approach to Affective Computing , 2008 .

[15]  Omar AlZoubi,et al.  Classification of EEG for Affect Recognition: An Adaptive Approach , 2009, Australasian Conference on Artificial Intelligence.

[16]  Ian H. Witten,et al.  Data mining - practical machine learning tools and techniques, Second Edition , 2005, The Morgan Kaufmann series in data management systems.

[17]  Rafael A. Calvo,et al.  Detecting Naturalistic Expressions of Nonbasic Affect Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[18]  Takashi Omori,et al.  ACE: Adaptive Classifiers-Ensemble System for Concept-Drifting Environments , 2005, Multiple Classifier Systems.

[19]  Geoff Holmes,et al.  New ensemble methods for evolving data streams , 2009, KDD.

[20]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[21]  Elisabeth André,et al.  Perception and Interactive Technologies, International Tutorial and Research Workshop, PIT 2006, Kloster Irsee, Germany, June 19-21, 2006, Proceedings , 2006, PIT.

[22]  David G. Stork,et al.  Pattern Classification , 1973 .

[23]  Robi Polikar,et al.  An Ensemble Approach for Incremental Learning in Nonstationary Environments , 2007, MCS.

[24]  Marcus A. Maloof,et al.  Dynamic weighted majority: a new ensemble method for tracking concept drift , 2003, Third IEEE International Conference on Data Mining.

[25]  Ludmila I. Kuncheva,et al.  Classifier Ensembles for Changing Environments , 2004, Multiple Classifier Systems.

[26]  Andrew R. Webb,et al.  Statistical Pattern Recognition , 1999 .

[27]  R. Lazarus Emotion and Adaptation , 1991 .

[28]  Rosalind W. Picard Affective Computing , 1997 .

[29]  J. Andreassi Psychophysiology: Human Behavior and Physiological Response , 1980 .

[30]  LastMark Online classification of nonstationary data streams , 2002 .

[31]  Ludmila I. Kuncheva,et al.  Multi-modal biometric emotion recognition using classifier ensembles , 2011, IEA/AIE'11.

[32]  Edwin Lughofer,et al.  Learning in Non-Stationary Environments , 2012 .

[33]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[34]  Roman Garnett,et al.  Sequential non-stationary dynamic classification with sparse feedback , 2010, Pattern Recognit..

[35]  Jason Williams,et al.  Emotion Recognition Using Bio-sensors: First Steps towards an Automatic System , 2004, ADS.

[36]  R. Polikar,et al.  Ensemble based systems in decision making , 2006, IEEE Circuits and Systems Magazine.

[37]  R. Levenson Autonomic Nervous System Differences among Emotions , 1992 .

[38]  Alfons O. Hamm,et al.  Motivational organization of emotions: Autonomic changes, cortical responses, and reflex modulation. , 2003 .

[39]  Richard J Davidson,et al.  Test-retest reliability of voluntary emotion regulation. , 2009, Psychophysiology.

[40]  Ludmila I. Kuncheva,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2004 .

[41]  K. H. Kim,et al.  Emotion recognition system using short-term monitoring of physiological signals , 2004, Medical and Biological Engineering and Computing.

[42]  Elisabeth André,et al.  Emotion Recognition Using Physiological and Speech Signal in Short-Term Observation , 2006, PIT.

[43]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[44]  P. Lang,et al.  International Affective Picture System (IAPS): Instruction Manual and Affective Ratings (Tech. Rep. No. A-4) , 1999 .

[45]  Mark Last,et al.  Online classification of nonstationary data streams , 2002, Intell. Data Anal..

[46]  Guillaume Chanel,et al.  Emotion Assessment: Arousal Evaluation Using EEG's and Peripheral Physiological Signals , 2006, MRCS.

[47]  Johannes Wagner,et al.  From Physiological Signals to Emotions: Implementing and Comparing Selected Methods for Feature Extraction and Classification , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[48]  J. Russell A circumplex model of affect. , 1980 .

[49]  Elisabeth André,et al.  Emotion recognition based on physiological changes in music listening , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  Jon Atli Benediktsson,et al.  Multiple Classifier Systems , 2015, Lecture Notes in Computer Science.

[51]  Stuart J. Russell,et al.  Online bagging and boosting , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[52]  Shigeo Abe DrEng Pattern Classification , 2001, Springer London.

[53]  Egon L. van den Broek,et al.  Unobtrusive Sensing of Emotions (USE) , 2009, J. Ambient Intell. Smart Environ..

[54]  Kagan Tumer,et al.  Classifier ensembles: Select real-world applications , 2008, Inf. Fusion.

[55]  P. Lang The emotion probe. Studies of motivation and attention. , 1995, The American psychologist.

[56]  Thomas Jürgensohn,et al.  Comparing Two Emotion Models for Deriving Affective States from Physiological Data , 2008, Affect and Emotion in Human-Computer Interaction.

[57]  Sylvia D. Kreibig,et al.  Autonomic nervous system activity in emotion: A review , 2010, Biological Psychology.

[58]  Plamen Angelov,et al.  Evolving Intelligent Systems: Methodology and Applications , 2010 .

[59]  Gerhard Widmer,et al.  Learning in the presence of concept drift and hidden contexts , 2004, Machine Learning.

[60]  Vasant Honavar,et al.  Learn++: an incremental learning algorithm for supervised neural networks , 2001, IEEE Trans. Syst. Man Cybern. Part C.

[61]  P. Ekman An argument for basic emotions , 1992 .

[62]  Emre Ertin,et al.  Continuous inference of psychological stress from sensory measurements collected in the natural environment , 2011, Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks.

[63]  M. Bradley,et al.  The International Affective Picture System (IAPS) in the study of emotion and attention. , 2007 .

[64]  L. Maier-Hein,et al.  Session independent non-audible speech recognition using surface electromyography , 2005, IEEE Workshop on Automatic Speech Recognition and Understanding, 2005..

[65]  P. Lang International Affective Picture System (IAPS) : Technical Manual and Affective Ratings , 1995 .