Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines

Synchrony refers to individuals' temporal coordination during social interactions. The analysis of this phenomenon is complex, requiring the perception and integration of multimodal communicative signals. The evaluation of synchrony has received multidisciplinary attention because of its role in early development, language learning, and social connection. Originally studied by developmental psychologists, synchrony has now captured the interest of researchers in such fields as social signal processing, robotics, and machine learning. This paper emphasizes the current questions asked by synchrony evaluation and the state-of-the-art related methods. First, we present definitions and functions of synchrony in youth and adulthood. Next, we review the noncomputational and computational approaches of annotating, evaluating, and modeling interactional synchrony. Finally, the current limitations and future research directions in the fields of developmental robotics, social robotics, and clinical studies are discussed.

[1]  W. S. Condon,et al.  A segmentation of behavior , 1967 .

[2]  A. Kendon Movement coordination in social interaction: some examples described. , 1970, Acta psychologica.

[3]  W. S. Condon,et al.  Neonate Movement Is Synchronized with Adult Speech: Interactional Participation and Language Acquisition , 1974, Science.

[4]  J. Cappella TALK AND SILENCE SEQUENCES IN INFORMAL CONVERSATIONS II , 1979 .

[5]  C. Trevarthen Communication and cooperation in early infancy: a description of primary intersubjectivity , 1979 .

[6]  B. Newman,et al.  Development Through Life: A Psychosocial Approach , 1979 .

[7]  J. Cappella,et al.  Talk and Silence Sequences in Informal Conversations III: Interspeaker Influence , 1981 .

[8]  M. Milgram,et al.  Probabilistic automata as a model for epigenesis of cellular networks. , 1983, Journal of theoretical biology.

[9]  R. Street Speech Convergence and Speech Evaluation in Fact-Finding Interviews , 1984 .

[10]  S. Garrod,et al.  Saying what you mean in dialogue: A study in conceptual and semantic co-ordination , 1987, Cognition.

[11]  R. Krauss,et al.  Dominance and accommodation in the conversational behaviours of same- and mixed-gender dyads. , 1988 .

[12]  Frank J. Bernieri,et al.  Synchrony, pseudosynchrony, and dissynchrony: Measuring the entrainment process in mother-infant interactions. , 1988 .

[13]  J. Reznick,et al.  Synchrony, pseudosynchrony, and dissynchrony: Measuring the entrainment process in mothernfant interactions. , 1988 .

[14]  D. Stern Le monde interpersonnel du nourrisson : une perspective psychanalytique et développementale , 1989 .

[15]  M. Turvey,et al.  Phase transitions and critical fluctuations in the visual coordination of rhythmic movements between people. , 1990, Journal of experimental psychology. Human perception and performance.

[16]  H. Giles,et al.  Accommodation theory: Communication, context, and consequence. , 1991 .

[17]  Frank J. Bernieri,et al.  Interpersonal coordination: Behavior matching and interactional synchrony. , 1991 .

[18]  S. Garrod,et al.  The Development of Dialogue Co-ordination Skills in Schoolchildren. , 1993 .

[19]  C L Webber,et al.  Dynamical assessment of physiological systems and states using recurrence plot strategies. , 1994, Journal of applied physiology.

[20]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[21]  R. Baron,et al.  Effects of Social and Physical Variables on Between-Person Visual Coordination , 1994 .

[22]  L. Lundqvist Facial EMG reactions to facial expressions: a case of facial emotional contagion? , 1995, Scandinavian journal of psychology.

[23]  G. Rizzolatti,et al.  Premotor cortex and the recognition of motor actions. , 1996, Brain research. Cognitive brain research.

[24]  R N Aslin,et al.  Statistical Learning by 8-Month-Old Infants , 1996, Science.

[25]  H. H. Clark,et al.  Conceptual pacts and lexical choice in conversation. , 1996, Journal of experimental psychology. Learning, memory, and cognition.

[26]  J. Cappella Behavioral and Judged Coordination in Adult Informal Social Interactions: Vocal and Kinesic Indicators , 1997 .

[27]  R. Schmidt,et al.  Evaluating the Dynamics of Unintended Interpersonal Coordination , 1997 .

[28]  P. Kuhl,et al.  Cross-language analysis of phonetic units in language addressed to infants. , 1997, Science.

[29]  K. Grammer,et al.  The Courtship Dance: Patterns of Nonverbal Synchronization in Opposite-Sex Encounters , 1998 .

[30]  R. Schmidt,et al.  A comparison of intra- and interpersonal interlimb coordination: coordination breakdowns and coupling strength. , 1998, Journal of experimental psychology. Human perception and performance.

[31]  J. Cassell,et al.  Conversation as a System Framework : Designing Embodied Conversational Agents , 1999 .

[32]  T. Chartrand,et al.  The chameleon effect: the perception-behavior link and social interaction. , 1999, Journal of personality and social psychology.

[33]  R. Schmidt,et al.  Self-organization of between-persons cooperative tasks and possible application to sport. , 1999 .

[34]  J. Nadel,et al.  Expectancies for social contingency in 2‐month‐olds , 1999 .

[35]  Alexandra A. Cleland,et al.  Syntactic co-ordination in dialogue , 2000, Cognition.

[36]  A. Bouhuys,et al.  Lack of coordination of nonverbal behaviour between patients and interviewers as a potential risk factor to depression recurrence: vulnerability accumulation in depression. , 2000, Journal of affective disorders.

[37]  P. Kuhl A new view of language acquisition. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[38]  M S Magnusson,et al.  Discovering hidden time patterns in behavior: T-patterns and their detection , 2000, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[39]  C. Trevarthen,et al.  Infant intersubjectivity: research, theory, and clinical applications. , 2001, Journal of child psychology and psychiatry, and allied disciplines.

[40]  S. Boker,et al.  Symmetry Building and Symmetry Breaking in Synchronized Movement , 2001 .

[41]  Justine Cassell,et al.  Human conversation as a system framework: designing embodied conversational agents , 2001 .

[42]  J. Decety,et al.  A common framework for perception and action: Neuroimaging evidence , 2001 .

[43]  A. Harrist,et al.  Dyadic Synchrony: Its Structure and Function in Children's Development. , 2002 .

[44]  S. Boker,et al.  Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series. , 2002, Psychological methods.

[45]  B. Granström,et al.  NATURAL TURN-TAKING NEEDS NO MANUAL : COMPUTATIONAL THEORY AND MODEL , FROM PERCEPTION TO ACTION , 2002 .

[46]  R. Feldman,et al.  Skin-to-Skin contact (Kangaroo care) promotes self-regulation in premature infants: sleep-wake cyclicity, arousal modulation, and sustained exploration. , 2002, Developmental psychology.

[47]  Joydeep Ghosh,et al.  Cluster Ensembles --- A Knowledge Reuse Framework for Combining Multiple Partitions , 2002, J. Mach. Learn. Res..

[48]  D. Stern Le monde interpersonnel du nourrisson , 2003 .

[49]  T. Chartrand,et al.  Using Nonconscious Behavioral Mimicry to Create Affiliation and Rapport , 2003, Psychological science.

[50]  K. Shockley,et al.  Mutual interpersonal postural constraints are involved in cooperative conversation. , 2003, Journal of experimental psychology. Human perception and performance.

[51]  R. Feldman,et al.  Direct and indirect effects of breast milk on the neurobehavioral and cognitive development of premature infants. , 2003, Developmental psychobiology.

[52]  Bar-Ilan,et al.  INFANT – MOTHER AND INFANT – FATHER SYNCHRONY : THE COREGULATION OF POSITIVE AROUSAL , 2003 .

[53]  Jean Carletta,et al.  A shallow model of backchannel continuers in spoken dialogue , 2003 .

[54]  M. Goldstein,et al.  Social interaction shapes babbling: Testing parallels between birdsong and speech , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[55]  R. Feldman,et al.  Mother-Child touch patterns in infant feeding disorders: relation to maternal, child, and environmental factors. , 2004, Journal of the American Academy of Child and Adolescent Psychiatry.

[56]  M. Rutter,et al.  Are there biological programming effects for psychological development? Findings from a study of Romanian adoptees. , 2004, Developmental psychology.

[57]  M. Pickering,et al.  Toward a mechanistic psychology of dialogue , 2004, Behavioral and Brain Sciences.

[58]  Michael T. Turvey,et al.  Phase-entrainment dynamics of visually coupled rhythmic movements , 1994, Biological Cybernetics.

[59]  R. Feldman,et al.  Parenting stress, infant emotion regulation, maternal sensitivity, and the cognitive development of triplets: a model for parent and child influences in a unique ecology. , 2004, Child development.

[60]  Michael T. Turvey,et al.  Frequency detuning of the phase entrainment dynamics of visually coupled rhythmic movements , 1995, Biological Cybernetics.

[61]  P. Kuhl Early language acquisition: cracking the speech code , 2004, Nature Reviews Neuroscience.

[62]  M. Pickering,et al.  Why is conversation so easy? , 2004, Trends in Cognitive Sciences.

[63]  G. K. Jonsson,et al.  Initial Interpersonal Attraction between Mixed-Sex Dyad and Movement Synchrony , 2005 .

[64]  J. Nadel,et al.  Experiencing contingency and agency : First step toward self-understanding in making a mind? , 2005 .

[65]  S. Duncan,et al.  Microanalysis of T-patterns . Analysis of Symmetry / Asymmetry in Social Interaction , 2005 .

[66]  C. Trevarthen,et al.  Disorganized rhythm and synchrony: Early signs of autism and Rett syndrome , 2005, Brain and Development.

[67]  L. Anolli The hidden structure of interaction : from neurons to culture patterns , 2005 .

[68]  Daniel C. Richardson,et al.  Looking To Understand: The Coupling Between Speakers' and Listeners' Eye Movements and Its Relationship to Discourse Comprehension , 2005, Cogn. Sci..

[69]  M. Plumet,et al.  Understanding the Functioning of Social Interaction with Autistic Children , 2005 .

[70]  A. Caspi,et al.  Maternal depression and children's antisocial behavior: nature and nurture effects. , 2005, Archives of general psychiatry.

[71]  Michael J. Richardson,et al.  Effects of visual and verbal interaction on unintentional interpersonal coordination. , 2005, Journal of experimental psychology. Human perception and performance.

[72]  A. Meltzoff,et al.  Imitation et développement humain : les premiers temps de la vie* , 2005 .

[73]  P. Perruchet,et al.  Implicit learning and statistical learning: one phenomenon, two approaches , 2006, Trends in Cognitive Sciences.

[74]  F. Ramseyer,et al.  Synchrony : A Core Concept for a Constructivist Approach to Psychotherapy , 2009 .

[75]  R. Feldman Parent-infant synchrony and the construction of shared timing; physiological precursors, developmental outcomes, and risk conditions. , 2007, Journal of child psychology and psychiatry, and allied disciplines.

[76]  Rajesh P. N. Rao,et al.  Imitation and Social Learning in Robots, Humans and Animals: A Bayesian model of imitation in infants and robots , 2007 .

[77]  Michael J. Richardson,et al.  Rocking together: dynamics of intentional and unintentional interpersonal coordination. , 2007, Human movement science.

[78]  Ken Prepin,et al.  Human–machine interaction as a model of machine–machine interaction: how to make machines interact as humans do , 2007, Adv. Robotics.

[79]  Ning Wang,et al.  Creating Rapport with Virtual Agents , 2007, IVA.

[80]  International Foundation for Autonomous Agents and MultiAgent Systems ( IFAAMAS ) , 2007 .

[81]  C. Heyes,et al.  Intact automatic imitation of human and robot actions in autism spectrum disorders , 2007, Proceedings of the Royal Society B: Biological Sciences.

[82]  Eric Keller,et al.  Prosodic and Gestural Expression of Interactional Agreement , 2007, COST 2102 Workshop.

[83]  J. Issartel,et al.  Unintended interpersonal co-ordination: “can we march to the beat of our own drum?” , 2007, Neuroscience Letters.

[84]  P. Kuhl Is speech learning 'gated' by the social brain? , 2007, Developmental science.

[85]  Michael J. Richardson,et al.  Articulatory constraints on interpersonal postural coordination. , 2007, Journal of experimental psychology. Human perception and performance.

[86]  C. Macrae,et al.  A case of hand waving: Action synchrony and person perception , 2008, Cognition.

[87]  Nick Campbell Multimodal Processing of Discourse Information; The Effect of Synchrony , 2008, 2008 Second International Symposium on Universal Communication.

[88]  J. Kelso,et al.  Social coordination dynamics: Measuring human bonding , 2008, Social neuroscience.

[89]  Louis-Philippe Morency,et al.  Predicting Listener Backchannels: A Probabilistic Multimodal Approach , 2008, IVA.

[90]  Michael Kipp Spatiotemporal Coding in ANVIL , 2008, LREC.

[91]  Daniel C. Richardson,et al.  Synchrony and swing in conversation: coordination, temporal dynamics and communication , 2008 .

[92]  Justine Cassell,et al.  Playing with virtual peers: bootstrapping contingent discourse in children with autism , 2008, ICLS.

[93]  Masashi Komori,et al.  Body Movement Synchrony in Psychotherapeutic Counseling: A Study Using the Video-Based Quantification Method , 2008, IEICE Trans. Inf. Syst..

[94]  Antonio Camurri,et al.  Emotional entrainment in music performance , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[95]  Maja Pantic,et al.  Social signal processing: Survey of an emerging domain , 2009, Image Vis. Comput..

[96]  Marek P. Michalowski,et al.  Keepon , 2009, Int. J. Soc. Robotics.

[97]  C. Macrae,et al.  The rhythm of rapport: Interpersonal synchrony and social perception , 2009 .

[98]  Julia Hirschberg,et al.  Backchannel-inviting cues in task-oriented dialogue , 2009, INTERSPEECH.

[99]  Scott S. Wiltermuth,et al.  Synchrony and Cooperation , 2009, Psychological science.

[100]  Fabian Ramseyer,et al.  Nonverbal Synchrony or Random Coincidence? How to Tell the Difference , 2009, COST 2102 Training School.

[101]  Marek P. Michalowski,et al.  Keepon : A Playful Robot for Research, Therapy, and Entertainment (Original Paper) , 2009 .

[102]  Reid G. Simmons,et al.  Rhythmic attention in child-robot dance play , 2009, RO-MAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication.

[103]  D. Schechter,et al.  Disturbances of attachment and parental psychopathology in early childhood. , 2009, Child and adolescent psychiatric clinics of North America.

[104]  R. Feldman,et al.  The Mother-Infant Feeding Relationship Across the First Year and the Development of Feeding Difficulties in Low-Risk Premature Infants. , 2009, Infancy : the official journal of the International Society on Infant Studies.

[105]  Kathleen T. Ashenfelter,et al.  Spatiotemporal symmetry and multifractal structure of head movements during dyadic conversation. , 2009, Journal of experimental psychology. Human perception and performance.

[106]  R. Feldman,et al.  Maternal representations, infant psychiatric status, and mother–child relationship in clinic-referred and non-referred infants , 2009, European Child & Adolescent Psychiatry.

[107]  Nick Campbell,et al.  An audio-visual approach to measuring discourse synchrony in multimodal conversation data , 2009, INTERSPEECH.

[108]  Terrence J Sejnowski,et al.  Foundations for a New Science of Learning , 2009, Science.

[109]  Philippe Gaussier,et al.  How an Agent Can Detect and Use Synchrony Parameter of Its Own Interaction with a Human? , 2009, COST 2102 Training School.

[110]  Olivier Oullier,et al.  Social Coordination, from the Perspective of Coordination Dynamics , 2009, Encyclopedia of Complexity and Systems Science.

[111]  Amy N. Dalton,et al.  Mimicry: Its ubiquity, importance, and functionality , 2009 .

[112]  Antonio Camurri,et al.  Toward a Real-Time Automated Measure of Empathy and Dominance , 2009, 2009 International Conference on Computational Science and Engineering.

[113]  Antonio Camurri,et al.  Measuring entrainment in small groups of musicians , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[114]  Shrikanth S. Narayanan,et al.  Predicting interruptions in dyadic spoken interactions , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[115]  M. Chetouani,et al.  What studies of family home movies can teach us about autistic infants: A literature review , 2010 .

[116]  Louis-Philippe Morency,et al.  Latent Mixture of Discriminative Experts for Multimodal Prediction Modeling , 2010, COLING.

[117]  Mohamed Chetouani,et al.  Multimodal coordination: exploring relevant features and measures , 2010, SSPW '10.

[118]  D. Lakens Movement synchrony and perceived entitativity , 2010 .

[119]  Jeffrey F. Cohn Advances in Behavioral Science Using Automated Facial Image Analysis and Synthesis [Social Sciences] , 2010, IEEE Signal Processing Magazine.

[120]  Anton Nijholt,et al.  Multimodal Embodied Mimicry in Interaction , 2010, COST 2102 Conference.

[121]  Dirk Heylen,et al.  Differences in head orientation behavior for speakers and listeners: An experiment in a virtual environment , 2010, TAP.

[122]  Paul Ruvolo,et al.  Applying machine learning to infant interaction: The development is in the details , 2010, Neural Networks.

[123]  Mohamed Chetouani,et al.  Understanding Parent-Infant Behaviors Using Non-negative Matrix Factorization , 2010, COST 2102 Training School.

[124]  C. Macrae,et al.  Moving memories: Behavioral synchrony and memory for self and others , 2010 .

[125]  Olac Fuentes,et al.  Dialog prediction for a general model of turn-taking , 2010, INTERSPEECH.

[126]  D. Cohen Probabilistic epigenesis: An alternative causal model for conduct disorders in children and adolescents , 2010, Neuroscience & Biobehavioral Reviews.

[127]  Antonio Camurri,et al.  A System for Real-Time Multimodal Analysis of Nonverbal Affective Social Interaction in User-Centric Media , 2010, IEEE Transactions on Multimedia.

[128]  Anton Nijholt,et al.  Measuring Multimodal Synchrony for Human-Computer Interaction , 2010, 2010 International Conference on Cyberworlds.

[129]  Giulio Sandini,et al.  In Press, Ieee Transactions on Autonomous Mental Development , 2010 .

[130]  Rajesh P. N. Rao,et al.  "Social" robots are psychological agents for infants: A test of gaze following , 2010, Neural Networks.

[131]  Louis-Philippe Morency,et al.  A multimodal end-of-turn prediction model: learning from parasocial consensus sampling , 2011, AAMAS.

[132]  Catherine Pelachaud,et al.  Shared Understanding and Synchrony Emergence - Synchrony as an Indice of the Exchange of Meaning between Dialog Partners , 2011, ICAART.

[133]  M. Chetouani,et al.  Do Parents Recognize Autistic Deviant Behavior Long before Diagnosis? Taking into Account Interaction Using Computational Methods , 2011, PloS one.

[134]  Jaime C. Acosta,et al.  Achieving rapport with turn-by-turn, user-responsive emotional coloring , 2011, Speech Commun..

[135]  Mohamed Chetouani,et al.  Characterization of coordination in an imitation task: human evaluation and automatically computable cues , 2011, ICMI '11.

[136]  Julia Hirschberg,et al.  Entrainment in Speech Preceding Backchannels , 2011, ACL.

[137]  Athanasios Katsamanis,et al.  An Analysis of PCA-Based Vocal Entrainment Measures in Married Couples' Affective Spoken Interactions , 2011, INTERSPEECH.

[138]  Joakim Gustafson,et al.  Predicting Speaker Changes and Listener Responses with and without Eye-Contact , 2011, INTERSPEECH.

[139]  Anton Nijholt,et al.  A Multimodal Database for Mimicry Analysis , 2011, ACII.

[140]  Anton Nijholt,et al.  Automatic visual mimicry expression analysis in interpersonal interaction , 2011, CVPR 2011 WORKSHOPS.

[141]  Anton Nijholt,et al.  Towards visual and vocal mimicry recognition in human-human interactions , 2011, 2011 IEEE International Conference on Systems, Man, and Cybernetics.

[142]  F. Ramseyer,et al.  Nonverbal synchrony in psychotherapy: coordinated body movement reflects relationship quality and outcome. , 2011, Journal of consulting and clinical psychology.

[143]  Daniel Gatica-Perez,et al.  Inferring truth from multiple annotators for social interaction analysis , 2011, NIPS 2011.

[144]  Julia Hirschberg,et al.  Pragmatic aspects of temporal accommodation in turn-taking , 2011, Journal of Pragmatics.

[145]  C. Crowell,et al.  The Clinical Use of Robots for Individuals with Autism Spectrum Disorders: A Critical Review. , 2012, Research in autism spectrum disorders.

[146]  付伶俐 打磨Using Language,倡导新理念 , 2014 .