Frame-differencing methods for measuring bodily synchrony in conversation

The study of interpersonal synchrony examines how interacting individuals grow to have similar behavior, cognition, and emotion in time. Many of the established methods of analyzing interpersonal synchrony are costly and time-consuming; the study of bodily synchrony has been especially laborious, traditionally requiring researchers to hand-code movement frame by frame. Because of this, researchers have been searching for more efficient alternatives for decades. Recently, some researchers (e.g., Nagaoka & Komori (IEICE Transactions on Information and Systems, 91(6), 1634–1640, 2008); Ramseyer & Tschacher, 2008) have applied computer science and computer vision techniques to create frame-differencing methods (FDMs) to simplify analyses. In this article, we provide a detailed presentation of one such FDM, created by modifying and adding to existing FDMs. The FDM that we present requires little programming experience or specialized equipment: Only a few lines of MATLAB code are required to execute an automated analysis of interpersonal synchrony. We provide sample code and demonstrate its use with an analysis of brief, friendly conversations; using linear mixed-effects models, the measure of interpersonal synchrony was found to be significantly predicted by time lag (p < .001) and by the interaction between time lag and measures of interpersonal liking (p < .001). This pattern of results fits with existing literature on synchrony. We discuss the current limitations and future directions for FDMs, including their use as part of a larger methodology for capturing and analyzing multimodal interaction.

[1]  Michael J. Richardson,et al.  Articulatory constraints on interpersonal postural coordination. , 2007, Journal of experimental psychology. Human perception and performance.

[2]  Fabian Ramseyer,et al.  Synchrony in dyadic psychotherapy sessions , 2008 .

[3]  S. Boker,et al.  Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series. , 2002, Psychological methods.

[4]  M. Pickering,et al.  Toward a mechanistic psychology of dialogue , 2004, Behavioral and Brain Sciences.

[5]  Daniel C. Richardson,et al.  Looking To Understand: The Coupling Between Speakers' and Listeners' Eye Movements and Its Relationship to Discourse Comprehension , 2005, Cogn. Sci..

[6]  Fabian Ramseyer,et al.  Video-based quantification of body movement during social interaction indicates the severity of negative symptoms in patients with schizophrenia , 2010, Schizophrenia Research.

[7]  Michael J. Richardson,et al.  Do birds of a feather move together? Group membership and behavioral synchrony , 2011, Experimental Brain Research.

[8]  Daniel S. Shaw,et al.  Mother–Son Positive Synchrony in Middle Childhood: Relation to Antisocial Behavior , 2003 .

[9]  Frank J. Bernieri,et al.  Synchrony, pseudosynchrony, and dissynchrony: Measuring the entrainment process in mother-infant interactions. , 1988 .

[10]  Daniel C. Richardson,et al.  Conversation, Gaze Coordination, and Beliefs About Visual Context , 2009, Cogn. Sci..

[11]  R. Baayen,et al.  Mixed-effects modeling with crossed random effects for subjects and items , 2008 .

[12]  Lauren H. Supplee,et al.  The correlates of dyadic synchrony in high-risk, low-income toddler boys. , 2006, Infant behavior & development.

[13]  Gina M. Caucci When I Move, You Move: Coordination in Conversation , 2011 .

[14]  Masashi Komori,et al.  Body Movement Synchrony in Psychotherapeutic Counseling: A Study Using the Video-Based Quantification Method , 2008, IEICE Trans. Inf. Syst..

[15]  K. Grammer,et al.  The Courtship Dance: Patterns of Nonverbal Synchronization in Opposite-Sex Encounters , 1998 .

[16]  Greg Welch,et al.  Motion Tracking: No Silver Bullet, but a Respectable Arsenal , 2002, IEEE Computer Graphics and Applications.

[17]  Elisabeth Dévière,et al.  Analyzing linguistic data: a practical introduction to statistics using R , 2009 .

[18]  Alexandra A. Cleland,et al.  The use of lexical and syntactic information in language production: Evidence from the priming of noun-phrase structure , 2003 .

[19]  F. Ramseyer,et al.  Nonverbal synchrony in psychotherapy: coordinated body movement reflects relationship quality and outcome. , 2011, Journal of consulting and clinical psychology.

[20]  T. Chartrand,et al.  The chameleon effect: the perception-behavior link and social interaction. , 1999, Journal of personality and social psychology.

[21]  Johanna D. Moore,et al.  Priming of Syntactic Rules in Task-Oriented Dialogue and Spontaneous Conversation , 2006 .

[22]  Rick Dale,et al.  Behavior Matching in Multimodal Communication Is Synchronized , 2012, Cogn. Sci..

[23]  Matt Huenerfauth,et al.  Collecting a Motion-Capture Corpus of American Sign Language for Data-Driven Generation Research , 2010, SLPAT@NAACL.

[24]  T. Chartrand,et al.  Using Nonconscious Behavioral Mimicry to Create Affiliation and Rapport , 2003, Psychological science.

[25]  Michael J. Richardson,et al.  Measuring the Dynamics of Interactional Synchrony , 2012 .

[26]  Erik Woody,et al.  Are we on the same wavelength? Interpersonal complementarity as shared cyclical patterns during interactions. , 2009, Journal of personality and social psychology.

[27]  M. Turvey,et al.  Phase transitions and critical fluctuations in the visual coordination of rhythmic movements between people. , 1990 .

[28]  Michael J. Richardson,et al.  Rocking together: dynamics of intentional and unintentional interpersonal coordination. , 2007, Human movement science.

[29]  K. Shockley,et al.  Mutual interpersonal postural constraints are involved in cooperative conversation. , 2003, Journal of experimental psychology. Human perception and performance.

[30]  D. Watson,et al.  Development and validation of brief measures of positive and negative affect: the PANAS scales. , 1988, Journal of personality and social psychology.

[31]  Piercarlo Valdesolo,et al.  Synchrony and the social tuning of compassion. , 2011, Emotion.

[32]  M. Turvey,et al.  Phase transitions and critical fluctuations in the visual coordination of rhythmic movements between people. , 1990, Journal of experimental psychology. Human perception and performance.

[33]  Matthew L. Jensen,et al.  Blob Analysis of the Head and Hands: A Method for Deception Detection , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[34]  Terry Marks-Tarlow,et al.  Simultaneity : temporal structures and observer perspectives , 2008 .

[35]  Matthew L. Jensen,et al.  Deception detection through automatic, unobtrusive analysis of nonverbal behavior , 2005, IEEE Intelligent Systems.

[36]  Frank J. Bernieri,et al.  Interactional Synchrony and Rapport: Measuring Synchrony in Displays Devoid of Sound and Facial Affect , 1994 .

[37]  K. Grammer,et al.  Fuzziness of nonverbal courtship communication unblurred by motion energy detection. , 1999, Journal of personality and social psychology.

[38]  Robert N. St. Clair,et al.  Language and social psychology , 1981 .

[39]  Spyros Kousidis,et al.  Monitoring Convergence of Temporal Features in Spontaneous Dialogue Speech , 2009 .

[40]  H. H. Clark,et al.  Conceptual pacts and lexical choice in conversation. , 1996, Journal of experimental psychology. Learning, memory, and cognition.

[41]  J. Pennebaker,et al.  Linguistic Style Matching in Social Interaction , 2002 .

[42]  W. S. Condon,et al.  Neonate Movement Is Synchronized with Adult Speech: Interactional Participation and Language Acquisition , 1974, Science.