User-State Sensing for Virtual Health Agents and TeleHealth Applications

Nonverbal behaviors play a crucial role in shaping outcomes in face-to-face clinical interactions. Experienced clinicians use nonverbals to foster rapport and "read" their clients to inform diagnoses. The rise of telemedicine and virtual health agents creates new opportunities, but it also strips away much of this nonverbal channel. Recent advances in low-cost computer vision and sensing technologies have the potential to address this challenge by learning to recognize nonverbal cues from large datasets of clinical interactions. These techniques can enhance both telemedicine and the emerging technology of virtual health agents. This article describes our current research in addressing these challenges in the domain of PTSD and depression screening for U.S. Veterans. We describe our general approach and report on our initial contribution: the creation of a large dataset of clinical interview data that facilitates the training of user-state sensing technology.

[1]  Judith A. Hall,et al.  Nonverbal behavior in clinician—patient interaction , 1995 .

[2]  L. Tickle-Degnen,et al.  The Nature of Rapport and Its Nonverbal Correlates , 1990 .

[3]  Terry L. Schell,et al.  All PTSD symptoms are highly associated with general distress: ramifications for the dysphoria symptom cluster. , 2010, Journal of abnormal psychology.

[4]  Patrick Kenny,et al.  SimCoach: an intelligent virtual human system for providing healthcare information and support , 2011 .

[5]  Stacy Marsella,et al.  Virtual Rapport , 2006, IVA.

[6]  Suzanne P. Weisband,et al.  Self disclosure on computer forms: meta-analysis and implications , 1996, CHI.

[7]  Rosalind W. Picard.  Affective Computing: From Laughter to IEEE , 2010 .

[8]  Laura M. Pfeifer,et al.  Relational Agents for Antipsychotic Medication Adherence , 2007 .

[9]  C. Darwin.  The Expression of the Emotions in Man and Animals , .

[10]  Pierre Philippot,et al.  Nonverbal Behavior in Clinical Settings , 2003 .

[11]  Jonathan Gratch,et al.  Virtual humans elicit socially anxious interactants' verbal self‐disclosure , 2010, Comput. Animat. Virtual Worlds.

[12]  J. Elhai,et al.  Testing whether posttraumatic stress disorder and major depressive disorder are similar or unique constructs. , 2011, Journal of anxiety disorders.

[13]  C. D. De Dreu,et al.  The interpersonal effects of anger and happiness in negotiations. , 2004, Journal of personality and social psychology.

[14]  Patrick Olivier,et al.  Proceedings of the 6th international conference on Intelligent Virtual Agents , 2006 .

[15]  J. Schelde,et al.  Major depression: behavioral parameters of depression and recovery. , 1998, The Journal of nervous and mental disease.

[16]  Richard L. Street,et al.  Signal Delay Effects on Rapport in Telepsychiatry , 2000, Cyberpsychology Behav. Soc. Netw..

[17]  Delroy L. Paulhus,et al.  Balanced Inventory of Desirable Responding , 2012 .

[18]  C. Spielberger,et al.  Manual for the State-Trait Anxiety Inventory , 1970 .

[19]  E. Blanchard,et al.  Psychometric properties of the PTSD Checklist (PCL). , 1996, Behaviour research and therapy.

[20]  P. Waxer.  Nonverbal cues for depression. , 1974, Journal of abnormal psychology.

[21]  J. Schelde.  Major depression: behavioral markers of depression and recovery. , 1998, The Journal of nervous and mental disease.

[22]  Robert Boice,et al.  Specification of nonverbal behaviors for clinical assessment , 1982 .

[23]  N. Ambady,et al.  Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. , 1992 .

[24]  D. Watson,et al.  Development and validation of brief measures of positive and negative affect: the PANAS scales. , 1988, Journal of personality and social psychology.

[25]  R. Spitzer,et al.  The PHQ-9 , 2001, Journal of General Internal Medicine.

[26]  S. Brunnhuber,et al.  Facial Expression and Experience of Emotions in Psychodynamic Interviews with Patients with PTSD in Comparison to Healthy Subjects , 2007, Psychopathology.

[27]  P. Ekman.  Universals and cultural differences in facial expressions of emotion. , 1972 .

[28]  Rena A. Menke.  Examining nonverbal shame markers among post-pregnancy women with maltreatment histories , 2011 .

[29]  M T McGuire,et al.  Nonverbal interaction of patients and therapists during psychiatric interviews. , 1982, Journal of abnormal psychology.

[30]  Sharon L. Oviatt,et al.  Perceptual user interfaces: multimodal interfaces that process what comes naturally , 2000, CACM.