Emotion Recognition From Gait Analyses: Current Research and Future Directions

Human gait refers to a daily motion that represents not only mobility, but it can also be used to identify the walker by either human observers or computers. Recent studies reveal that gait even conveys information about the walker's emotion. Individuals in different emotion states may show different gait patterns. The mapping between various emotions and gait patterns provides a new source for automated emotion recognition. Compared to traditional emotion detection biometrics, such as facial expression, speech and physiological parameters, gait is remotely observable, more difficult to imitate, and requires less cooperation from the subject. These advantages make gait a promising source for emotion detection. This article reviews current research on gait-based emotion detection, particularly on how gait parameters can be affected by different emotion states and how the emotion states can be recognized through distinct gait patterns. We focus on the detailed methods and techniques applied in the whole process of emotion recognition: data collection, preprocessing, and classification. At last, we discuss possible future developments of efficient and effective gait-based emotion recognition using the state of the art techniques on intelligent computation and big data.

[1]  R. Laban,et al.  The mastery of movement , 1950 .

[2]  P. Ekman,et al.  Head and body cues in the judgment of emotion: a reformulation. , 1967, Perceptual and motor skills.

[3]  J. Russell,et al.  Evidence for a three-factor theory of emotions , 1977 .

[4]  J. Cutting,et al.  Recognizing friends by their walk: Gait perception without familiarity cues , 1977 .

[5]  J. Montepare,et al.  The identification of emotions from gait information , 1987 .

[6]  S. Bouisset,et al.  Biomechanical study of the programming of anticipatory postural adjustments associated with voluntary movement. , 1987, Journal of biomechanics.

[7]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[8]  R. D. Walk,et al.  Perception of the smile and other emotions of the body and face at different distances , 1988 .

[9]  G V Cochran,et al.  Dynamic electromyography. I. Numerical representation using principal component analysis , 1990, Journal of orthopaedic research : official publication of the Orthopaedic Research Society.

[10]  Stéphane Mallat,et al.  Singularity detection and processing with wavelets , 1992, IEEE Trans. Inf. Theory.

[11]  D. Winter,et al.  Predicting muscle forces in gait from EMG signals and musculotendon kinematics. , 1992, Journal of electromyography and kinesiology : official journal of the International Society of Electrophysiological Kinesiology.

[12]  John G. Proakis,et al.  Digital Signal Processing: Principles, Algorithms, and Applications , 1992 .

[13]  J. Gage Gait analysis. An essential tool in the treatment of cerebral palsy. , 1993, Clinical orthopaedics and related research.

[14]  P. Ekman Facial expression and emotion. , 1993, The American psychologist.

[15]  Jon D. Morris Observations: SAM: The Self-Assessment Manikin An Efficient Cross-Cultural Measurement Of Emotional Response 1 , 1995 .

[16]  P. Allard,et al.  Functional gait asymmetry in able-bodied subjects , 1997 .

[17]  K. Deluzio,et al.  Principal component models of knee kinematics and kinetics: Normal vs. pathological gait patterns , 1997 .

[18]  Peter J. Lang,et al.  Attention and Orienting : Sensory and Motivational Processes , 1997 .

[19]  S. Olney,et al.  Multivariate examination of data from gait analysis of persons with stroke. , 1998, Physical therapy.

[20]  S. Asfour,et al.  Discrete wavelet transform: a tool in smoothing kinematic data. , 1999, Journal of biomechanics.

[21]  T. Dalgleish,et al.  Handbook of cognition and emotion , 1999 .

[22]  S. Chaiken,et al.  Personality and Social Psychology Bulle- Tin Chen, Bargh / Consequences of Automatic Evaluation Immediate Behavioral Predispositions to Approach or Avoid the Stimulus , 2022 .

[23]  M. Lemke,et al.  Spatiotemporal gait patterns during over ground locomotion in major depression compared with healthy controls. , 2000, Journal of psychiatric research.

[24]  T Chau,et al.  A review of analytical techniques for gait data. Part 1: Fuzzy, statistical and fractal methods. , 2001, Gait & posture.

[25]  D. Armstrong Neuropathology of Rett syndrome. , 2002, Mental retardation and developmental disabilities research reviews.

[26]  Beat Fasel,et al.  Automati Fa ial Expression Analysis: A Survey , 1999 .

[27]  Rama Chellappa,et al.  Identification of humans using gait , 2004, IEEE Transactions on Image Processing.

[28]  Mark C. Coulson Attributing Emotion to Static Body Postures: Recognition Accuracy, Confusions, and Viewpoint Dependence , 2004 .

[29]  D. Hatzinakos,et al.  Gait recognition: a challenging signal processing technology for biometric identification , 2005, IEEE Signal Processing Magazine.

[30]  Sam J. Maglio,et al.  Emotional category data on images from the international affective picture system , 2005, Behavior research methods.

[31]  M. Shiffrar,et al.  Recognizing people from their movement. , 2005, Journal of experimental psychology. Human perception and performance.

[32]  Sudeep Sarkar,et al.  The humanID gait challenge problem: data sets, performance, and analysis , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Xiaolan Fu,et al.  The Reliability and Validity of the Chinese Version of Abbreviated PAD Emotion Scales , 2005, ACII.

[34]  M. Perc The dynamics of human gait , 2005 .

[35]  M. N. Nyan,et al.  Classification of gait patterns in the time-frequency domain. , 2006, Journal of biomechanics.

[36]  R. Shiavi,et al.  Representing and clustering electromyographic gait patterns with multivariate techniques , 1981, Medical and Biological Engineering and Computing.

[37]  K. H. Kim,et al.  Emotion recognition system using short-term monitoring of physiological signals , 2004, Medical and Biological Engineering and Computing.

[38]  R. Sorrentino,et al.  Electronic Filter Simulation & Design , 2007 .

[39]  Ginevra Castellano,et al.  Recognising Human Emotions from Body Movement and Gesture Dynamics , 2007, ACII.

[40]  Lars Omlor,et al.  Extraction of spatio-temporal primitives of emotional body expressions , 2006, Neurocomputing.

[41]  J. Jankovic Parkinson’s disease: clinical features and diagnosis , 2008, Journal of Neurology, Neurosurgery, and Psychiatry.

[42]  D. Janssen,et al.  Recognition of Emotions in Gait Patterns by Means of Artificial Neural Nets , 2008 .

[43]  Hatice Gunes,et al.  Automatic Temporal Segment Detection and Affect Recognition From Face and Body Display , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[44]  Claire L. Roether,et al.  Critical features for the perception of emotion from gait. , 2009, Journal of vision.

[45]  N. Troje,et al.  Embodiment of Sadness and Depression—Gait Patterns Associated With Dysphoric Mood , 2009, Psychosomatic medicine.

[46]  K. Scherer,et al.  Bodily expression of emotion , 2009 .

[47]  Michelle Karg,et al.  Recognition of Affect Based on Gait Patterns , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[48]  Gentiane Venture Human characterization and emotion characterization from gait , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[49]  Lianhong Cai,et al.  Facial Expression Synthesis Based on Emotion Dimensions for Affective Talking Avatar , 2010, Modeling Machine Emotions for Realizing Intelligence.

[50]  Thierry Gélat,et al.  Gait initiation is affected during emotional conflict , 2011, Neuroscience Letters.

[51]  Fakhri Karray,et al.  Survey on speech emotion recognition: Features, classification schemes, and databases , 2011, Pattern Recognit..

[52]  P. Beek,et al.  Organization of voluntary stepping in response to emotion-inducing pictures. , 2011, Gait & posture.

[53]  Zhengyou Zhang,et al.  Microsoft Kinect Sensor and Its Effect , 2012, IEEE Multim..

[54]  Kelly M. Naugle,et al.  Emotional state affects gait initiation in individuals with Parkinson’s disease , 2012, Cognitive, affective & behavioral neuroscience.

[55]  T. Flash,et al.  Expression of emotion in the kinematics of locomotion , 2012, Experimental Brain Research.

[56]  Elizabeth A. Crane,et al.  Effort-Shape and kinematic assessment of bodily expression of emotion during gait. , 2012, Human movement science.

[57]  Atsuo Takanishi,et al.  The influences of emotional intensity for happiness and sadness on walking , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[58]  Yiorgos Chrysanthou,et al.  The Next Big Thing Automatic Emotion Recognition Based on Body Movement Analysis A Survey , 2014 .

[59]  Begonya Garcia-Zapirain,et al.  Gait Analysis Methods: An Overview of Wearable and Non-Wearable Systems, Highlighting Clinical Applications , 2014, Sensors.

[60]  Gentiane Venture,et al.  Recognizing Emotions Conveyed by Human Gait , 2014, International Journal of Social Robotics.

[61]  P. White Appraisal Theory , 2015 .

[62]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[63]  Andrea Cavallaro,et al.  Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[64]  Thierry Gélat,et al.  Reaction time in gait initiation depends on the time available for affective processing , 2015, Neuroscience Letters.

[65]  P. Beek,et al.  Biomechanical organization of gait initiation depends on the timing of affective processing. , 2015, Gait & posture.

[66]  Gu Eon Kang,et al.  Emotional influences on sit-to-walk in healthy young adults. , 2015, Human movement science.

[67]  C. Ioana,et al.  Wavelet-based characterization of gait signal for neurological abnormalities. , 2015, Gait & posture.

[68]  Gu Eon Kang,et al.  The effect of emotion on movement smoothness during gait in healthy young adults. , 2016, Journal of biomechanics.

[69]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[70]  Kaat Desloovere,et al.  Gait analysis: clinical facts. , 2016, European journal of physical and rehabilitation medicine.

[71]  Tingshao Zhu,et al.  Emotion recognition based on customized smart bracelet with built-in accelerometer , 2016, PeerJ.

[72]  Shun Li,et al.  Emotion recognition using Kinect motion capture data of human gaits , 2016, PeerJ.

[73]  Bin Hu,et al.  Self-esteem recognition based on gait pattern using Kinect. , 2017, Gait & posture.

[74]  Xiaogang Wang,et al.  A Comprehensive Study on Cross-View Gait Based Human Identification with Deep CNNs , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[75]  Juan Song,et al.  Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM , 2017, IEEE Access.

[76]  Fazel Naghdy,et al.  Automatic Affect Perception Based on Body Gait and Posture: A Survey , 2017, International Journal of Social Robotics.

[77]  Pan Hui,et al.  Emotion Recognition through Gait on Mobile Devices , 2018, 2018 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops).

[78]  Laura Avanzino,et al.  Relationships between gait and emotion in Parkinson's disease: A narrative review. , 2018, Gait & posture.

[79]  Sergio Escalera,et al.  RGB-D-based Human Motion Recognition with Deep Learning: A Survey , 2017, Comput. Vis. Image Underst..

[80]  Tingshao Zhu,et al.  Identifying Emotions from Non-Contact Gaits Information Based on Microsoft Kinects , 2018, IEEE Transactions on Affective Computing.

[81]  Juan C. Quiroz,et al.  Emotion Recognition Using Smart Watch Sensor Data: Mixed-Design Study , 2018, JMIR mental health.

[82]  Marina L. Gavrilova,et al.  Score and Rank-Level Fusion for Emotion Recognition Using Genetic Algorithm , 2018, 2018 IEEE 17th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC).

[83]  Christian Kroos,et al.  Not all is noticed: Kinematic cues of emotion-specific gait. , 2017, Human movement science.