Multi-Sensory Emotion Recognition with Speech and Facial Expression

of Dissertation Submitted to the Graduate School of The University of Southern Mississippi in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy

[1]  Guanglin Li,et al.  Principal Components Analysis Preprocessing for Improved Classification Accuracies in Pattern-Recognition-Based Myoelectric Control , 2009, IEEE Transactions on Biomedical Engineering.

[2]  Bruce Edmonds,et al.  Socially Intelligent Agents: Creating Relationships With Computers And Robots , 2013 .

[3]  Ashok Samal,et al.  Human Face Detection Using Silhouettes , 1995, Int. J. Pattern Recognit. Artif. Intell..

[4]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[5]  Benoit Huet,et al.  Bimodal Emotion Recognition , 2010, ICSR.

[6]  Chil-Woo Lee,et al.  Detection and pose estimation of human face with synthesized image models , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[7]  Björn W. Schuller,et al.  Segmenting into Adequate Units for Automatic Recognition of Emotion-Related Episodes: A Speech-Based Approach , 2010, Adv. Hum. Comput. Interact..

[8]  Björn W. Schuller,et al.  Abandoning emotion classes - towards continuous emotion recognition with modelling of long-range dependencies , 2008, INTERSPEECH.

[9]  Pietro Perona,et al.  Probabilistic affine invariants for recognition , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[10]  Valery A. Petrushin.  Creating Emotion Recognition Agents for Speech Signal , 2002 .

[11]  Björn W. Schuller,et al.  The INTERSPEECH 2009 emotion challenge , 2009, INTERSPEECH.

[12]  Elmar Nöth,et al.  We are not amused - but how do you know? user states in a multi-modal dialogue system , 2003, INTERSPEECH.

[13]  Shrikanth S. Narayanan,et al.  Combining categorical and primitives-based emotion recognition , 2006, 2006 14th European Signal Processing Conference.

[14]  Loïc Kessous,et al.  The relevance of feature type for the automatic classification of emotional user states: low level descriptors and functionals , 2007, INTERSPEECH.

[15]  Zhengyou Zhang,et al.  A Survey of Recent Advances in Face Detection , 2010 .

[16]  Thomas S. Huang,et al.  Emotion Recognition from Facial Expressions using Multilevel HMM , 2000 .

[17]  Jun Ohya,et al.  Recognizing multiple persons' facial expressions using HMM based on automatic extraction of significant frames from image sequences , 1997, Proceedings of International Conference on Image Processing.

[18]  Björn W. Schuller,et al.  Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge , 2011, Speech Commun..

[19]  Zhongzhe Xiao,et al.  Hierarchical Classification of Emotional Speech , 2007 .

[20]  Maja J. Mataric,et al.  Evaluating evaluators: a case study in understanding the benefits and pitfalls of multi-evaluator modeling , 2009, INTERSPEECH.

[21]  Vidhyasaharan Sethu,et al.  Pitch contour parameterisation based on linear stylisation for emotion recognition , 2009, INTERSPEECH.

[22]  Niels da Vitoria Lobo,et al.  Face detection using templates , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[23]  Paul Lamere,et al.  Sphinx-4: a flexible open source framework for speech recognition , 2004 .

[24]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[25]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Marian Stewart Bartlett,et al.  Classifying Facial Actions , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  S. Tomkins,et al.  Affect Imagery Consciousness: The Positive Affects , 1963 .

[28]  T. Dalgleish.  Basic Emotions , 2004 .

[29]  Alessandra Russo,et al.  Speech Emotion Classification Using Machine Learning Algorithms , 2008, 2008 IEEE International Conference on Semantic Computing.

[30]  Loïc Kessous,et al.  Whodunnit - Searching for the most important feature types signalling emotion-related user states in speech , 2011, Comput. Speech Lang..

[31]  Jiucang Hao,et al.  Emotion recognition by speech signals , 2003, INTERSPEECH.

[32]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  Stephan Hamann,et al.  Mapping discrete and dimensional emotions onto the brain: controversies and consensus , 2012, Trends in Cognitive Sciences.

[34]  Makoto Nagao,et al.  Line extraction and pattern detection in a photograph , 1969, Pattern Recognit..

[35]  Michael C. Burl,et al.  Finding faces in cluttered scenes using random labeled graph matching , 1995, Proceedings of IEEE International Conference on Computer Vision.

[36]  Thomas S. Huang,et al.  Human face detection in a complex background , 1994, Pattern Recognit..

[37]  Andries van Dam.  Beyond WIMP , 2000, IEEE Computer Graphics and Applications.

[38]  Björn W. Schuller,et al.  Multimodal emotion recognition in audiovisual communication , 2002, Proceedings. IEEE International Conference on Multimedia and Expo.

[39]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[40]  Klaus R. Scherer,et al.  Vocal communication of emotion , 2000 .

[41]  Larry S. Davis,et al.  Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Beat Fasel,et al.  Automati Fa ial Expression Analysis: A Survey , 1999 .

[43]  Loïc Kessous,et al.  Multimodal emotion recognition from expressive faces, body gestures and speech , 2007, AIAI.

[44]  C. Darwin.  The Expression of the Emotions in Man and Animals , .

[45]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[46]  Aastha Joshi.  Speech Emotion Recognition Using Combined Features of HMM & SVM Algorithm , 2013 .

[47]  Vicki Bruce.  What the human face tells the human mind: some challenges for the robot-human interface , 1993, Adv. Robotics.

[48]  Takeo Kanade,et al.  Probabilistic modeling of local appearance and spatial relationships for object recognition , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[49]  Astrid Paeschke,et al.  A database of German emotional speech , 2005, INTERSPEECH.

[50]  K. Scherer,et al.  Conscious emotional experience emerges as a function of multilevel, appraisal-driven response synchronization , 2008, Consciousness and Cognition.

[51]  Valery A. Petrushin,et al.  EMOTION IN SPEECH: RECOGNITION AND APPLICATION TO CALL CENTERS , 1999 .

[52]  Pierre-Yves Oudeyer,et al.  The production and recognition of emotions in speech: features and algorithms , 2003, Int. J. Hum. Comput. Stud..

[53]  R. Plutchik,et al.  Emotion: Theory, Research, and Experience. Vol. 1. Theories of Emotion , 1981 .

[54]  Kenji Mase,et al.  Recognition of Facial Expression from Optical Flow , 1991 .

[55]  Ling Guan,et al.  Recognizing Human Emotional State From Audiovisual Signals , 2008, IEEE Transactions on Multimedia.

[56]  Robert I. Damper,et al.  Emotion recognition from speech using extended feature selection and a simple classifier , 2009, INTERSPEECH.

[57]  Lori Lamel,et al.  Challenges in real-life emotion annotation and machine learning based detection , 2005, Neural Networks.

[58]  A. Tanju Erdem,et al.  INTERSPEECH 2009 Emotion Recognition Challenge evaluation , 2010, 2010 IEEE 18th Signal Processing and Communications Applications Conference.

[59]  Krzysztof Slot,et al.  Emotion recognition in speech signal using emotion-extracting binary decision trees , 2007 .

[60]  Takeo Kanade,et al.  Neural Network-Based Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[61]  Elisabeth André,et al.  Comparing Feature Sets for Acted and Spontaneous Speech in View of Automatic Emotion Recognition , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[62]  Frank Dellaert,et al.  Recognizing emotion in speech , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[63]  Chun Chen,et al.  Manifolds Based Emotion Recognition in Speech , 2007, ROCLING/IJCLCLP.

[64]  Milan Sonka,et al.  Image Processing, Analysis and Machine Vision , 1993, Springer US.

[65]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[66]  Constantine Kotropoulos,et al.  Fast sequential floating forward selection applied to emotional speech features estimated on DES and SUSAS data collections , 2006, 2006 14th European Signal Processing Conference.

[67]  Mauridhi Hery Purnomo,et al.  Facial Emotional Expressions of Li fe-like Character Based on Text Classifier and Fuzzy Logic , 2011 .

[68]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[69]  Ioannis Pitas,et al.  The eNTERFACE’05 Audio-Visual Emotion Database , 2006, 22nd International Conference on Data Engineering Workshops (ICDEW'06).

[70]  Federico Girosi,et al.  Training support vector machines: an application to face detection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[71]  Joan Claudi Socoró,et al.  GTM-URL contribution to the INTERSPEECH 2009 emotion challenge , 2009, INTERSPEECH.

[72]  Timothy F. Cootes,et al.  An Automatic Face Identification System Using Flexible Appearance Models , 1994, BMVC.

[73]  Constantine Kotropoulos,et al.  Emotional speech recognition: Resources, features, and methods , 2006, Speech Commun..

[74]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[75]  Michael J. Lyons,et al.  Classifying facial attributes using a 2-D Gabor wavelet representation and discriminant analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[76]  Tomaso A. Poggio,et al.  Example-Based Learning for View-Based Human Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[77]  Michael J. Black,et al.  Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion , 1995, Proceedings of IEEE International Conference on Computer Vision.

[78]  Tiago H. Falk,et al.  Long-term spectro-temporal information for improved automatic speech emotion classification , 2008, INTERSPEECH.

[79]  Elmar Nöth,et al.  “You Stupid Tin Box” - Children Interacting with the AIBO Robot: A Cross-linguistic Emotional Speech Corpus , 2004, LREC.

[80]  Young Joon Ahn,et al.  An approximation of circular arcs by quartic Bézier curves , 2007, Comput. Aided Des..

[81]  Florian Schiel,et al.  The SmartKom Multimodal Corpus at BAS , 2002, LREC.

[82]  Shrikanth S. Narayanan,et al.  Toward detecting emotions in spoken dialogs , 2005, IEEE Transactions on Speech and Audio Processing.

[83]  Johannes Wagner,et al.  Automatic Recognition of Emotions from Speech: A Review of the Literature and Recommendations for Practical Realisation , 2008, Affect and Emotion in Human-Computer Interaction.

[84]  อนิรุธ สืบสิงห์,et al.  Data Mining Practical Machine Learning Tools and Techniques , 2014 .

[85]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[86]  Shohei Kato,et al.  Bayesian-Based Inference of Dialogist's Emotion for Sensitivity Robots , 2007, RO-MAN 2007 - The 16th IEEE International Symposium on Robot and Human Interactive Communication.

[87]  Michael S. Lew,et al.  Information theoretic view-based and modular face detection , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[88]  Jörn Ostermann,et al.  Animation of synthetic faces in MPEG-4 , 1998, Proceedings Computer Animation '98 (Cat. No.98EX169).

[89]  Javier Macías Guarasa,et al.  Acoustic emotion recognition using dynamic Bayesian networks and multi-space distributions , 2009, INTERSPEECH.

[90]  R. Plutchik,et al.  Theories of emotion , 1980 .

[91]  Thurid Vogt,et al.  Real-time automatic emotion recognition from speech , 2010 .

[92]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[93]  Uday B. Desai,et al.  Finding faces in photographs , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[94]  Björn W. Schuller,et al.  Balancing spoken content adaptation and unit length in the recognition of emotion and interest , 2008, INTERSPEECH.

[95]  Johannes Wagner,et al.  A Systematic Comparison of Different HMM Designs for Emotion Recognition from Acted and Spontaneous Speech , 2007, ACII.

[96]  Sharon L. Oviatt,et al.  Multimodal Integration - A Statistical View , 1999, IEEE Trans. Multim..

[97]  A. Ortony,et al.  What's basic about basic emotions? , 1990, Psychological review.

[98]  Guoyin Wang,et al.  Speech Emotion Recognition Based on Rough Set and SVM , 2006, 2006 5th IEEE International Conference on Cognitive Informatics.

[99]  Carlos Busso,et al.  IEMOCAP: interactive emotional dyadic motion capture database , 2008, Lang. Resour. Evaluation.

[100]  Thomas S. Huang,et al.  Face detection with information-based maximum discrimination , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[101]  Bin Yang,et al.  Cascaded emotion classification via psychological emotion dimensions using a large set of voice quality parameters , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[102]  M. Mehta,et al.  MULTIMODAL INPUT FUSION IN HUMAN-COMPUTER INTERACTION On the Example of the NICE Project , 2003 .

[103]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[104]  Thierry Pun,et al.  Multimodal Emotion Recognition in Response to Videos , 2012, IEEE Transactions on Affective Computing.

[105]  Rosalind W. Picard,et al.  Classical and novel discriminant features for affect recognition from speech , 2005, INTERSPEECH.

[106]  Elisabeth André,et al.  Improving Automatic Emotion Recognition from Speech via Gender Differentiaion , 2006, LREC.

[107]  L. Rothkrantz,et al.  Toward an affect-sensitive multimodal human-computer interaction , 2003, Proc. IEEE.

[108]  Chloé Clavel,et al.  Detection and Analysis of Abnormal Situations Through Fear-Type Acoustic Manifestations , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[109]  Rosalind W. Picard,et al.  Modeling drivers' speech under stress , 2003, Speech Commun..

[110]  Zhigang Deng,et al.  Emotion recognition based on phoneme classes , 2004, INTERSPEECH.

[111]  Julia Sidorova,et al.  Speech Emotion Recognition With TGI+.2 Classifier , 2009, EACL.

[112]  Narendra Ahuja,et al.  Detecting Faces in Images: A Survey , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[113]  P. Ekman.  Emotion in the human face , 1982 .

[114]  Anna Wierzbicka,et al.  Talking about emotions : semantics, culture, and cognition , 1992 .

[115]  Roberto Cipolla,et al.  A probabilistic framework for perceptual grouping of features for human face detection , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[116]  Timothy F. Cootes,et al.  A unified approach to coding and interpreting face images , 1995, Proceedings of IEEE International Conference on Computer Vision.

[117]  K. Scherer,et al.  Appraisal processes in emotion: Theory, methods, research. , 2001 .

[118]  Kjell Elenius,et al.  Automatic recognition of anger in spontaneous speech , 2008, INTERSPEECH.

[119]  P. Ekman.  An argument for basic emotions , 1992 .

[120]  Björn W. Schuller,et al.  Speaker independent emotion recognition by early fusion of acoustic and linguistic features within ensembles , 2005, INTERSPEECH.

[121]  Colin Grubb.  Multimodal Emotion Recognition , 2013 .

[122]  Björn Schuller,et al.  Opensmile: the munich versatile and fast open-source audio feature extractor , 2010, ACM Multimedia.

[123]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).