A demonstration of the perception system in SimSensei, a virtual human application for healthcare interviews

We present the SimSensei system, a fully automatic virtual agent that conducts interviews to assess indicators of psychological distress. With this demo, we focus our attention on the perception part of the system, a multimodal framework which captures and analyzes user state behavior for both behavioral understanding and interactional purposes. We will demonstrate real-time user state sensing as a part of the SimSensei architecture and discuss how this technology enabled automatic analysis of behaviors related to psychological distress.

[1]  Louis-Philippe Morency,et al.  It's only a computer: Virtual humans increase willingness to disclose , 2014, Comput. Hum. Behav..

[2]  Louis-Philippe Morency,et al.  Automatic nonverbal behavior indicators of depression and PTSD: the effect of gender , 2014, Journal on Multimodal User Interfaces.

[3]  David DeVault,et al.  The Distress Analysis Interview Corpus of human and computer interviews , 2014, LREC.

[4]  Kallirroi Georgila,et al.  Verbal indicators of psychological distress in interactive dialogue with a virtual human , 2013, SIGDIAL Conference.

[5]  Albert A. Rizzo,et al.  User-State Sensing for Virtual Health Agents and TeleHealth Applications , 2013, MMVR.

[6]  Judee K. Burgoon,et al.  Nonverbal communication theories of interaction adaptation , 2010 .

[7]  Ning Wang,et al.  Using Social Agents to Explore Theories of Rapport and Emotional Resonance , 2014 .

[8]  Rosalind W. Picard,et al.  Establishing the computer-patient working alliance in automated health behavior change interventions. , 2005, Patient education and counseling.

[9]  Kallirroi Georgila,et al.  SimSensei kiosk: a virtual human interviewer for healthcare decision support , 2014, AAMAS.

[10]  J. Cohn,et al.  Impact of depression on response to comedy: a dynamic facial coding analysis. , 2007, Journal of abnormal psychology.

[11]  Yuyu Xu,et al.  Perception Markup Language: Towards a Standardized Representation of Perceived Nonverbal Behaviors , 2012, IVA.

[12]  Anton Leuski,et al.  All Together Now - Introducing the Virtual Human Toolkit , 2013, IVA.

[13]  Albert A. Rizzo,et al.  Automatic audiovisual behavior descriptors for psychological disorder analysis , 2014, Image Vis. Comput..

[14]  Catherine Pelachaud,et al.  The TARDIS Framework: Intelligent Virtual Agents for Social Coaching in Job Interviews , 2013, Advances in Computer Entertainment.

[15]  Louis-Philippe Morency,et al.  Investigating voice quality as a speaker-independent indicator of depression and PTSD , 2013, INTERSPEECH.