Changes in verbal and nonverbal conversational behavior in long-term interaction

We present an empirical investigation of conversational behavior in dyadic interaction spanning multiple conversations, in the context of a developing interpersonal relationship between a health counselor and her clients. Using a longitudinal video corpus of behavior change counseling conversations, we show systematic changes in verbal and nonverbal behavior during greetings (within the first minute of conversations). Both the number of prior conversations and self-reported assessments of the strength of the interpersonal relationship are predictive of changes in verbal and nonverbal behavior. We present a model and implementation of nonverbal behavior generation for conversational agents that incorporates these findings, and discuss how the results can be applied to multimodal recognition of conversational behavior over time.

[1]  Alastair J. Gill,et al.  Coordination in Conversation and Rapport , 2007 .

[2]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[3]  J. Garske,et al.  Relation of the therapeutic alliance with outcome and other variables: a meta-analytic review. , 2000, Journal of consulting and clinical psychology.

[4]  N. Breslow.  Extra‐Poisson Variation in Log‐Linear Models , 1984 .

[5]  E. Goffman.  Relations in Public: Microstudies of the Public Order , 1971 .

[6]  E. Bordin,et al.  Theory and research on the therapeutic working alliance: New directions. , 1994 .

[7]  Martyn Plummer,et al.  JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling , 2003 .

[8]  Hans-Peter Seidel,et al.  Annotated New Text Engine Animation Animation Lexicon Animation Gesture Profiles MR : . . . JL : . . . Gesture Generation Video Annotated Gesture Script , 2007 .

[9]  Justine Cassell,et al.  BEAT: the Behavior Expression Animation Toolkit , 2001, Life-like characters.

[10]  Klaus P. Schneider.  Small talk : analysing phatic discourse , 1988 .

[11]  Kurt Gray,et al.  Psychological Inquiry: An International Journal for the Advancement of Psychological Theory , 2012 .

[12]  Timothy W. Bickmore,et al.  Posture, Relationship, and Discourse Structure - Models of Nonverbal Behavior for Long-Term Interaction , 2011, IVA.

[13]  Linda Tickle-Degnen,et al.  Changes in Nonverbal Behavior During the Development of Therapeutic Relationships. , 2003 .

[14]  P. Waxer,et al.  Nonverbal cues for anxiety: an examination of emotional leakage. , 1977, Journal of abnormal psychology.

[15]  Michael Kipp,et al.  ANVIL - a generic annotation tool for multimodal dialogue , 2001, INTERSPEECH.

[16]  I. Altman,et al.  Social penetration: The development of interpersonal relationships , 1973 .

[17]  A. Abele,et al.  Functions of gaze in social interaction: Communication and monitoring , 1986 .

[18]  Michael Smithson,et al.  A better lemon squeezer? Maximum-likelihood regression with beta-distributed dependent variables. , 2006, Psychological methods.

[19]  Charles E. McCulloch,et al.  Generalized Linear Mixed Models , 2005 .

[20]  Linda Tickle-Degnen,et al.  Group rapport and nonverbal behavior. , 1987 .

[21]  Stacy Marsella,et al.  Nonverbal Behavior Generator for Embodied Conversational Agents , 2006, IVA.

[22]  S. Planalp,et al.  Friends' and Acquaintances' Conversations I: Perceived Differences , 1992 .

[23]  Stacy Marsella,et al.  Virtual Rapport , 2006, IVA.

[24]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[25]  T. Bickmore.  Relational agents : effecting change through human-computer relationships , 2003 .

[26]  Timothy W. Bickmore,et al.  Modeling Behavioral Manifestations of Coordination and Rapport over Multiple Conversations - Speaking Rate as a Relational Indicator for a Virtual Agent , 2010, IVA.

[27]  J. Garske,et al.  Relation of the therapeutic alliance with outcome and other variables: a meta-analytic review. , 2000, Journal of consulting and clinical psychology.

[28]  Stefan Kopp,et al.  Towards integrated microplanning of language and iconic gesture for multimodal output , 2004, ICMI '04.

[29]  C. McCulloch,et al.  Generalized Linear Mixed Models , 2005 .

[30]  J. Gillaspy,et al.  Development and validation of a revised short version of the working alliance inventory , 2006 .

[31]  D. Rubin,et al.  Inference from Iterative Simulation Using Multiple Sequences , 1992 .

[32]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[33]  Pierre Philippot,et al.  Nonverbal Behavior in Clinical Settings , 2003 .

[34]  P. Ekman,et al.  The Duchenne smile: emotional expression and brain physiology. II. , 1990, Journal of personality and social psychology.

[35]  L. Tickle-Degnen,et al.  The Nature of Rapport and Its Nonverbal Correlates , 1990 .