Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses

The relationship between nonverbal behavior and severity of depression was investigated by following depressed participants over the course of treatment and video recording a series of clinical interviews. Facial expressions and head pose were analyzed from video using manual and automatic systems. Both systems were highly consistent for FACS action units (AUs) and showed similar effects for change over time in depression severity. When symptom severity was high, participants made fewer affiliative facial expressions (AUs 12 and 15) and more non-affiliative facial expressions (AU 14). Participants also exhibited diminished head motion (i.e., amplitude and velocity) when symptom severity was high. These results are consistent with the Social Withdrawal hypothesis: that depressed individuals use nonverbal behavior to maintain or increase interpersonal distance. As individuals recover, they send more signals indicating a willingness to affiliate. The finding that automatic facial expression analysis was both consistent with manual coding and revealed the same pattern of findings suggests that automatic facial expression analysis may be ready to relieve the burden of manual coding in behavioral and clinical science.

[1]  Paul Ekman,et al.  Facial expression analysis , 2008, Scholarpedia.

[2]  E. Klinger,et al.  Consequences of commitment to and disengagement from incentives. , 1975 .

[3]  M. Strauss,et al.  Diminished response to pleasant stimuli by depressed women. , 2001, Journal of abnormal psychology.

[4]  J. Gross,et al.  Emotion context insensitivity in major depressive disorder. , 2005, Journal of abnormal psychology.

[5]  J. Gross,et al.  The tie that binds? Coherence among emotion experience, behavior, and physiology. , 2005, Emotion.

[6]  B. Everitt,et al.  Statistical methods for rates and proportions , 1973 .

[7]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[8]  S. Shapiro,et al.  An Analysis of Variance Test for Normality (Complete Samples) , 1965 .

[9]  Wade Junek,et al.  Mind Reading: The Interactive Guide to Emotions , 2007 .

[10]  M. Strauss,et al.  Subjective and expressive emotional responses in depression. , 1997, Journal of affective disorders.

[11]  D. Shapiro,et al.  Reduced facial expression and social context in major depression: discrepancies between facial muscle activity and self-reported emotion , 2000, Psychiatry Research.

[12]  Gwen Littlewort,et al.  Automatic Recognition of Facial Actions in Spontaneous Expressions , 2006, J. Multim..

[13]  M. First,et al.  User's guide for the structured clinical interview for DSM-IV axis I disorders : SCID-I : clinical version , 1997 .

[14]  James Price Dillard,et al.  The Interactional Theory of Depression: A Meta-Analysis of the Research Literature , 1992 .

[15]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[16]  Fernando De la Torre,et al.  Detecting depression from facial actions and vocal prosody , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[17]  A. Troisi,et al.  Gender differences in depression: an ethological study of nonverbal behavior during interviews. , 1999, Journal of psychiatric research.

[18]  M. Katsikitis,et al.  A Controlled Quantitative Study of Facial Expression in Parkinson's Disease and Depression , 1991, The Journal of nervous and mental disease.

[20]  W. Iacono,et al.  Neurobehavioral aspects of affective disorders. , 1989, Annual review of psychology.

[21]  Daniel S. Messinger,et al.  A framework for automated measurement of the intensity of non-posed Facial Action Units , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[22]  Sridha Sridharan,et al.  In the Pursuit of Effective Affective Computing: The Relationship Between Features and Registration , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[23]  J. Macher,et al.  Facial expressiveness in patients with schizophrenia compared to depressed patients and nonpatient comparison subjects. , 2005, The American journal of psychiatry.

[24]  Lawrence Cayton,et al.  Algorithms for manifold learning , 2005 .

[25]  Simon Lucey,et al.  Temporal coordination of head motion in couples with history of interpersonal violence , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[26]  B. Renneberg,et al.  Facial expression of emotions in borderline personality disorder and depression. , 2005, Journal of behavior therapy and experimental psychiatry.

[27]  J. Gross,et al.  Sadness and amusement reactivity differentially predict concurrent and prospective functioning in major depressive disorder. , 2002, Emotion.

[28]  G. Bower.  Mood and memory. , 1981, The American psychologist.

[29]  G. Klerman,et al.  Interpersonal psychotherapy of depression , 1984 .

[30]  Alison J. Longley,et al.  Depression is an adaptation. , 2001, Archives of general psychiatry.

[31]  C. Mathers,et al.  Projections of Global Mortality and Burden of Disease from 2002 to 2030 , 2006, PLoS medicine.

[32]  D. Watson,et al.  Tripartite model of anxiety and depression: psychometric evidence and taxonomic implications. , 1991, Journal of abnormal psychology.

[33]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[34]  Frank Schneider,et al.  Computer-based analysis of facial action in schizophrenic and depressed patients , 1990, European Archives of Psychiatry and Clinical Neuroscience.

[35]  Vlad I. Morariu,et al.  Expression , 2015, Principles of Molecular Virology.

[36]  K. Fletcher,et al.  Handbook of Psychiatric Measures , 2002 .

[37]  J. Cohn,et al.  Spontaneous facial expression in a small group can be automatically measured: An initial demonstration , 2010, Behavior research methods.

[38]  S. Sakamoto,et al.  Polygraphic Evaluation of Laughing and Smiling in Schizophrenic and Depressive Patients , 1997, Perceptual and motor skills.

[39]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[40]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[41]  Jeffrey B. Henriques,et al.  Decreased responsiveness to reward in depression , 2000 .

[42]  Fernando De la Torre,et al.  Facial Expression Analysis , 2011, Visual Analysis of Humans.

[43]  A. J. Fridlund.  Human Facial Expression: An Evolutionary View , 1994 .

[44]  Chih-Jen Lin,et al.  A Practical Guide to Support Vector Classication , 2008 .

[45]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  C. Segrin.  Social skills deficits associated with depression. , 2000, Clinical psychology review.

[47]  Dc Washington.  Diagnostic and Statistical Manual of Mental Disorders, 4th Ed. , 1994 .

[48]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[49]  J. Borod,et al.  Social functioning and facial emotional expression in neurological and psychiatric disorders. , 1998, Applied neuropsychology.

[50]  K. Fletcher,et al.  Handbook of Psychiatric Measures, 2nd Edition , 2008 .

[51]  M. Hamilton,et al.  Development of a rating scale for primary depressive illness. , 1967, The British journal of social and clinical psychology.

[52]  J. Schelde.  Major depression: behavioral markers of depression and recovery. , 1998, The Journal of nervous and mental disease.

[53]  Fernando De la Torre,et al.  Dynamic cascades with bidirectional bootstrapping for spontaneous facial action unit detection , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[54]  Samuel R. Friedman,et al.  Depression: Clinical, Experimental, and Theoretical Aspects , 1968 .

[55]  J. Rabe-Jabłońska,et al.  [Affective disorders in the fourth edition of the classification of mental disorders prepared by the American Psychiatric Association -- diagnostic and statistical manual of mental disorders]. , 1993, Psychiatria polska.

[56]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[57]  N. Freedman,et al.  Chapter 8 – The Analysis of Movement Behavior During the Clinical Interview1 , 1972 .

[58]  Lauren M. Bylsma,et al.  A meta-analysis of emotional reactivity in major depressive disorder. , 2008, Clinical psychology review.

[59]  John C. Markowitz,et al.  Treatment and Prevention of Depression , 2002, Psychological science in the public interest : a journal of the American Psychological Society.

[60]  J. Coyne,et al.  Toward an interactional description of depression. , 1976, Psychiatry.

[61]  J. Cohn,et al.  Impact of depression on response to comedy: a dynamic facial coding analysis. , 2007, Journal of abnormal psychology.

[62]  Janet B W Williams.  Diagnostic and Statistical Manual of Mental Disorders , 2013 .

[63]  L Sloman,et al.  Evolved mechanisms in depression: the role and interaction of attachment and social rank in depression. , 2003, Journal of affective disorders.

[64]  A K Dixon,et al.  Ethological Studies in Animals and Man, Their Use in Psychiatry , 1989, Pharmacopsychiatry.

[65]  Gwen Littlewort,et al.  Toward Practical Smile Detection , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[66]  Roland Göcke,et al.  Can body expressions contribute to automatic depression analysis? , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[67]  Takeo Kanade,et al.  Facial Expression Analysis , 2011, AMFG.

[68]  I K Fodor,et al.  A Survey of Dimension Reduction Techniques , 2002 .

[69]  Albert A. Rizzo,et al.  Automatic behavior descriptors for psychological disorder analysis , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[70]  B. Knutson.  Facial expressions of emotion influence interpersonal trait inferences , 1996 .

[71]  N. Allen,et al.  The social risk hypothesis of depressed mood: evolutionary, psychosocial, and neurobiological perspectives. , 2003, Psychological bulletin.

[72]  I. Jones,et al.  Some Nonverbal Aspects of Depression and Schizophrenia Occurring during the Interview , 1979, The Journal of nervous and mental disease.

[73]  Nico H. Frijda,et al.  Expression, emotion, neither, or both: Review of A. Fridlund, Facial expression of emotion , 1995 .

[74]  I. Gotlib,et al.  Further evidence for the cultural norm hypothesis: positive emotion in depressed and control European American and Asian American women. , 2010, Cultural diversity & ethnic minority psychology.

[75]  W. Grove.  Statistical Methods for Rates and Proportions, 2nd ed , 1981 .

[76]  Reginald B. Adams,et al.  Who may frown and who should smile? Dominance, affiliation, and the display of happiness and anger , 2005 .

[77]  H. Ellgring.  Nonverbal communication in depression , 1989 .

[78]  L. Fossi,et al.  The Ethological Approach to the Assessment of Depressive Disorders , 1984, The Journal of nervous and mental disease.

[79]  H P Hirsbrunner,et al.  Analyzing nonverbal behavior in depression. , 1983, Journal of abnormal psychology.

[80]  D Hvizdos.  The tie that binds. , 2000, The American journal of nursing.

[81]  H. Berenbaum,et al.  Emotional experience and expression in schizophrenia and depression. , 1992, Journal of abnormal psychology.

[82]  Simon Baker,et al.  Active Appearance Models Revisited , 2004, International Journal of Computer Vision.

[83]  D. Keltner,et al.  Embarrassment: its distinct form and appeasement functions. , 1997, Psychological bulletin.

[84]  M. First,et al.  Structured clinical interview for DSM-IV axis II personality disorders : SCID-II , 1997 .

[85]  J. Schelde.  Major depression: behavioral parameters of depression and recovery. , 1998, The Journal of nervous and mental disease.

[86]  W. Gaebel,et al.  Facial expressivity in the course of schizophrenia and depression , 2004, European Archives of Psychiatry and Clinical Neuroscience.

[87]  R. Levenson,et al.  The effects of depression on the emotional responses of Spanish-speaking Latinas. , 2003, Cultural diversity & ethnic minority psychology.

[88]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[89]  U. Hess,et al.  The Influence of Facial Emotion Displays, Gender, and Ethnicity on Judgments of Dominance and Affiliation , 2000 .

[90]  P. Lewinsohn,et al.  The functional relation between depression and problematic interpersonal behavior. , 1980, Journal of abnormal psychology.

[91]  F. Schmidt,et al.  Linking "big" personality traits to anxiety, depressive, and substance use disorders: a meta-analysis. , 2010, Psychological bulletin.

[92]  G. Juckel,et al.  Kinematical analysis of emotionally induced facial expressions: a novel tool to investigate hypomimia in patients suffering from depression , 2004, Journal of Neurology, Neurosurgery & Psychiatry.

[93]  S. Baron-Cohen,et al.  Mind Reading: The Interactive Guide to Emotions , 2003 .

[94]  Catharine E. Fairbairn,et al.  The effects of alcohol on the emotional displays of Whites in interracial groups. , 2013, Emotion.