Detection and Computational Analysis of Psychological Signals Using a Virtual Human Interviewing Agent

It has long been recognized that facial expressions, body posture/gestures and vocal parameters play an important role in human communication and the implicit signalling of emotion. Recent advances in low cost computer vision and behavioral sensing technologies can now be applied to the process of making meaningful inferences as to user state when a person interacts with a computational device. Effective use of this additive information could serve to promote human interaction with virtual human (VH) agents that may enhance diagnostic assessment. The same technology could also be leveraged to improve engagement in teletherapy approaches between remote patients and care providers. This paper will focus on our current research in these areas within the DARPA-funded “Detection and Computational Analysis of Psychological Signals” project, with specific attention to the SimSensei application use case. SimSensei is a virtual human interaction platform that is able to sense and interpret real-time audiovisual behavioral signals from users interacting with the system. It is specifically designed for health care support and leverages years of virtual human research and development at USC-ICT. The platform enables an engaging face-to-face interaction where the virtual human automatically reacts to the state and inferred intent of the user through analysis of behavioral signals gleaned from facial expressions, body gestures and vocal parameters. Akin to how non-verbal behavioral signals have an impact on human to human interaction and communication, SimSensei aims to capture and infer from user non-verbal communication to improve engagement between a VH and a user. The system can also quantify and interpret sensed behavioral signals longitudinally that can be used to inform diagnostic assessment within a clinical context.

[1]  C. Spielberger,et al.  Manual for the State-Trait Anxiety Inventory , 1970 .

[2]  Stacy Marsella,et al.  How virtual reality training can win friends and influence people , 2013 .

[3]  Terry L. Schell,et al.  All PTSD symptoms are highly associated with general distress: ramifications for the dysphoria symptom cluster. , 2010, Journal of abnormal psychology.

[4]  Ronald R. Holden,et al.  Comparisons among the Holden Psychological Screening Inventory (HPSI), the Brief Symptom Inventory (BSI), and the Balanced Inventory of Desirable Responding (BIDR) , 2000, Assessment.

[5]  Peter Robinson,et al.  3D Constrained Local Model for rigid and non-rigid facial tracking , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Patrick Kenny,et al.  SimCoach: an intelligent virtual human system for providing healthcare information and support , 2011 .

[7]  C. Darwin.  The Expression of the Emotions in Man and Animals , .

[8]  J. Schelde.  Major depression: behavioral markers of depression and recovery. , 1998, The Journal of nervous and mental disease.

[9]  James A. Russell,et al.  The psychology of facial expression: What does a facial expression mean? , 1997 .

[10]  Kallirroi Georgila,et al.  Verbal indicators of psychological distress in interactive dialogue with a virtual human , 2013, SIGDIAL Conference.

[11]  John E. Perez,et al.  Nonverbal Social Skills and Psychopathology , 2003 .

[12]  J. Schelde.  Major depression: behavioral parameters of depression and recovery. , 1998, The Journal of nervous and mental disease.

[13]  Alex Pentland,et al.  Honest Signals - How They Shape Our World , 2008 .

[14]  Louis-Philippe Morency,et al.  Automatic Nonverbal Behavior Indicators of Depression and PTSD: Exploring Gender Differences , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[15]  D. Lazer,et al.  Using reality mining to improve public health and medicine. , 2009, Studies in health technology and informatics.

[16]  Fernando De la Torre,et al.  Detecting depression from facial actions and vocal prosody , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[17]  E. Blanchard,et al.  Psychometric properties of the PTSD Checklist (PCL). , 1996, Behaviour research and therapy.

[18]  P. Ekman,et al.  The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding , 1969 .

[19]  Louis-Philippe Morency,et al.  It's only a computer: Virtual humans increase willingness to disclose , 2014, Comput. Hum. Behav..

[20]  J. Pennebaker,et al.  Language use of depressed and depression-vulnerable college students , 2004 .

[21]  Christopher R. Erbes,et al.  The predictive validity of the PTSD Checklist in a nonclinical sample of combat-exposed National Guard troops. , 2012, Psychological assessment.

[22]  Suzanne P. Weisband,et al.  Self disclosure on computer forms: meta-analysis and implications , 1996, CHI.

[23]  David DeVault,et al.  The Distress Analysis Interview Corpus of human and computer interviews , 2014, LREC.

[24]  Jonathan Gratch,et al.  Virtual humans elicit socially anxious interactants' verbal self‐disclosure , 2010, Comput. Animat. Virtual Worlds.

[25]  R. Spitzer,et al.  The PHQ-9 , 2001, Journal of General Internal Medicine.

[26]  Judith A. Hall,et al.  Nonverbal behavior in clinician—patient interaction , 1995 .

[27]  D. Watson,et al.  Development and validation of brief measures of positive and negative affect: the PANAS scales. , 1988, Journal of personality and social psychology.

[28]  Lauren M. Bylsma,et al.  A meta-analysis of emotional reactivity in major depressive disorder. , 2008, Clinical psychology review.

[29]  J. Elhai,et al.  Testing whether posttraumatic stress disorder and major depressive disorder are similar or unique constructs. , 2011, Journal of anxiety disorders.

[30]  Louis-Philippe Morency,et al.  Investigating voice quality as a speaker-independent indicator of depression and PTSD , 2013, INTERSPEECH.

[31]  C. Darwin,et al.  The Expression of the Emotions in Man and Animals , 1956 .

[32]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[33]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[34]  Matthew Turk,et al.  Perceptual user interfaces , 2000 .

[35]  Albert A. Rizzo,et al.  Automatic behavior descriptors for psychological disorder analysis , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[36]  S. Brunnhuber,et al.  Facial Expression and Experience of Emotions in Psychodynamic Interviews with Patients with PTSD in Comparison to Healthy Subjects , 2007, Psychopathology.

[37]  Jonathan Gratch,et al.  Socially Anxious People Reveal More Personal Information with Virtual Counselors That Talk about Themselves using Intimate Human Back Stories , 2012, Annual Review of Cybertherapy and Telemedicine.

[38]  Rena A. Menke.  Examining nonverbal shame markers among post-pregnancy women with maltreatment histories , 2011 .

[39]  M. Antony,et al.  The State-Trait Anxiety Inventory, Trait version: structure and content re-examined. , 1998, Behaviour research and therapy.

[40]  Kallirroi Georgila,et al.  SimSensei kiosk: a virtual human interviewer for healthcare decision support , 2014, AAMAS.

[41]  M T McGuire,et al.  Nonverbal interaction of patients and therapists during psychiatric interviews. , 1982, Journal of abnormal psychology.

[42]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[43]  Javier R. Movellan,et al.  Generalized adaptive view-based appearance model: Integrated framework for monocular head pose estimation , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[44]  P. Waxer.  Nonverbal cues for depression. , 1974, Journal of abnormal psychology.

[45]  Gwen Littlewort,et al.  Toward Practical Smile Detection , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  J. Russell,et al.  Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant. , 1999, Journal of personality and social psychology.