Interactionism in language: from neural networks to bodies to dyads

ABSTRACT In a science of language, it can be useful to partition different formats of linguistic information into different categories, such as phonetics, phonology, semantics, and syntax. However, when the actual phenomena of language processing cross those boundaries and blur those lines, it can become difficult to understand how these different formats of linguistic information maintain their integrity while engaging in complex interactions with one another. For example, if the function of a cortical network that is known to process phonetics is immediately taking into account contextual influences from a cortical network that is known to process semantics, then it seems clear that this “phonetic cortical network” is doing more than just phonetics. In the neuroscience and cognitive science of language, the scope of analysis where different formats of linguistic information are seen to interact reveals a wide array of context effects in almost every possible direction. When one expands the scope of analysis to include nonlinguistic sensory modalities, such as vision and action, research is showing that even those lines are getting blurred. Visual perception and motor movement appear to influence various aspects of language processing in real time. As this scope of analysis expands further still, research is showing that two human brains and bodies exhibit various forms of synchrony or coordination with one another during natural conversation. Interactionism at all these levels of analysis poses a challenge to traditional frameworks that treat different components of language, perception, and action as operating via domain specific computations.

[1]  Pyeong Whan Cho,et al.  Incremental parsing in a continuous dynamical system: sentence processing in Gradient Symbolic Computation , 2017 .

[2]  R. Ilmoniemi,et al.  Functional links between motor and language systems , 2005, The European journal of neuroscience.

[3]  S. Goldinger Echoes of echoes? An episodic theory of lexical access. , 1998, Psychological review.

[4]  Lawrence W. Barsalou,et al.  Are Automatic Conceptual Cores the Gold Standard of Semantic Processing? The Context-Dependence of Spatial Meaning in Grounded Congruency Effects , 2015, Cogn. Sci..

[5]  Christopher T. Kello,et al.  “How do humans make sense?” multiscale dynamics and emergent meaning , 2017, New Ideas in Psychology.

[6]  J. Obleser,et al.  Expectancy constraints in degraded speech modulate the language comprehension network. , 2010, Cerebral cortex.

[7]  John-Dylan Haynes,et al.  Content-specific coordination of listeners' to speakers' EEG during communication , 2012, Front. Hum. Neurosci..

[8]  Lawrence W. Barsalou,et al.  Language comprehension: Archival memory or preparation for situated action? , 1999 .

[9]  S. Scott,et al.  Functional Integration across Brain Regions Improves Speech Perception under Adverse Listening Conditions , 2007, The Journal of Neuroscience.

[10]  Marta Kutas,et al.  Comprehending how visual context influences incremental sentence processing: Insights from ERPs and picture-sentence verification. , 2011, Psychophysiology.

[11]  D. Kemmerer,et al.  The Two-Level Theory of verb meaning: An approach to integrating the semantics of action with the mirror neuron system , 2010, Brain and Language.

[12]  G. Buzsáki,et al.  Neuronal Oscillations in Cortical Networks , 2004, Science.

[13]  D. Plaut Double dissociation without modularity: evidence from connectionist neuropsychology. , 1995, Journal of clinical and experimental neuropsychology.

[14]  Z. Harris,et al.  Foundations of Language , 1940 .

[15]  Rolf A. Zwaan,et al.  PSYCHOLOGICAL SCIENCE Research Article THE EFFECT OF IMPLIED ORIENTATION DERIVED FROM VERBAL CONTEXT ON PICTURE RECOGNITION , 2022 .

[16]  Roger M. Cooper,et al.  The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. , 1974 .

[17]  Rick Dale,et al.  Behavior Matching in Multimodal Communication Is Synchronized , 2012, Cogn. Sci..

[18]  Roel M. Willems,et al.  Embodied Cognition: Taking the Next Step , 2012, Front. Psychology.

[19]  L P Shapiro,et al.  "How to milk a coat:" the effects of semantic and acoustic information on phoneme categorization. , 1998, The Journal of the Acoustical Society of America.

[20]  G. Altmann,et al.  The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing , 2007 .

[21]  Herbert Schriefers,et al.  Discourse-level semantic coherence influences beta oscillatory dynamics and the N400 during sentence comprehension , 2017 .

[22]  Matthew W. Crocker,et al.  The Coordinated Interplay of Scene, Utterance, and World Knowledge: Evidence From Eye Tracking , 2006, Cogn. Sci..

[23]  Mante S. Nieuwland,et al.  When Peanuts Fall in Love: N400 Evidence for the Power of Discourse , 2005, Journal of Cognitive Neuroscience.

[24]  Julie C. Sedivy,et al.  Achieving incremental semantic interpretation through contextual representation , 1999, Cognition.

[25]  Gavan Lintern,et al.  Dynamic patterns: The self-organization of brain and behavior , 1997, Complex.

[26]  Michael J. Spivey,et al.  Toward a New Scientific Visualization for the Language Sciences , 2012, Inf..

[27]  Michael J. Richardson,et al.  Coordination dynamics in a socially situated nervous system , 2012, Front. Hum. Neurosci..

[28]  T. Chartrand,et al.  The chameleon effect: the perception-behavior link and social interaction. , 1999, Journal of personality and social psychology.

[29]  D Norris,et al.  Merging information in speech recognition: Feedback is never necessary , 2000, Behavioral and Brain Sciences.

[30]  M. Tanenhaus,et al.  Modeling the Influence of Thematic Fit (and Other Constraints) in On-line Sentence Comprehension , 1998 .

[31]  Rolf A. Zwaan,et al.  Language Comprehenders Mentally Represent the Shapes of Objects , 2002, Psychological science.

[32]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[33]  Paul D. Allopenna,et al.  Tracking the Time Course of Spoken Word Recognition Using Eye Movements: Evidence for Continuous Mapping Models , 1998 .

[34]  Christopher T. Kello,et al.  Soft-assembly of sensorimotor function. , 2009, Nonlinear dynamics, psychology, and life sciences.

[35]  David W Gow,et al.  Sentential influences on acoustic-phonetic processing: a Granger causality analysis of multimodal imaging data , 2016, Language, cognition and neuroscience.

[36]  Corticospinal excitability during the processing of handwritten and typed words and non-words , 2017, Neuroscience Letters.

[37]  J. Fodor The Modularity of mind. An essay on faculty psychology , 1986 .

[38]  Colin M. Brown,et al.  Anticipating upcoming words in discourse: evidence from ERPs and reading times. , 2005, Journal of experimental psychology. Learning, memory, and cognition.

[39]  J. Trueswell,et al.  Interpreting pronouns and demonstratives in Finnish: Evidence for a form-specific approach to reference resolution , 2008 .

[40]  Michael J. Spivey,et al.  An inclusive account of mind across spatiotemporal scales of cognition , 2017 .

[41]  Michael J. Spivey,et al.  Probabilistic Constraint Satisfaction at the Lexical/Phonetic Interface: Evidence for Gradient Effects of Within-Category VOT on Lexical Access , 2003, Journal of psycholinguistic research.

[42]  D. Norris,et al.  No lexical–prelexical feedback during speech perception or: Is it time to stop playing those Christmas tapes? , 2009 .

[43]  G. A. Miller,et al.  The intelligibility of speech as a function of the context of the test materials. , 1951, Journal of experimental psychology.

[44]  Thomas F Münte,et al.  Cerebral Cortex Advance Access published July 21, 2007 Visual Scenes Trigger Immediate Syntactic Reanalysis: Evidence from ERPs during Situated Spoken Comprehension , 2022 .

[45]  The Module Mistake , 2003, Cortex.

[46]  Michael L. Anderson,et al.  Eroding the boundaries of cognition: implications of embodiment(1). , 2012, Topics in cognitive science.

[47]  W. Marslen-Wilson Functional parallelism in spoken word-recognition , 1987, Cognition.

[48]  Daniel C. Richardson,et al.  The Art of Conversation Is Coordination , 2007, Psychological science.

[49]  J. Fodor The Modularity of mind. An essay on faculty psychology , 1986 .

[50]  R. M. Warren Perceptual Restoration of Missing Speech Sounds , 1970, Science.

[51]  M. Kutas,et al.  Reading senseless sentences: brain potentials reflect semantic incongruity. , 1980, Science.

[52]  J. Fodor,et al.  The Modularity of Mind: An Essay on Faculty Psychology , 1984 .

[53]  Susan M. Garnsey,et al.  Semantic Influences On Parsing: Use of Thematic Role Information in Syntactic Ambiguity Resolution , 1994 .

[54]  Michael K. Tanenhaus,et al.  Dynamical models of sentence processing , 1999, Cogn. Sci..

[55]  C. Connine Constraints on interactive processes in auditory word recognition: The role of sentence context , 1987 .

[56]  Michael J. Spivey,et al.  Syntactic ambiguity resolution in discourse: modeling the effects of referential context and lexical frequency. , 1998, Journal of experimental psychology. Learning, memory, and cognition.

[57]  K. Shockley,et al.  Mutual interpersonal postural constraints are involved in cooperative conversation. , 2003, Journal of experimental psychology. Human perception and performance.

[58]  Colin M. Brown,et al.  Semantic Integration in Sentences and Discourse: Evidence from the N400 , 1999, Journal of Cognitive Neuroscience.

[59]  Chris Arney Sync: The Emerging Science of Spontaneous Order , 2007 .

[60]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[61]  Matthew Goldrick,et al.  Optimization and Quantization in Gradient Symbol Systems: A Framework for Integrating the Continuous and the Discrete in Cognition , 2014, Cogn. Sci..

[62]  Z. Harris,et al.  Foundations of language , 1941 .

[63]  Michael J. Spivey,et al.  Interactivity of language , 2018, Lang. Linguistics Compass.

[64]  J. V. van Berkum,et al.  How robust is the language architecture? The case of mood , 2013, Front. Psychol..

[65]  J. Trueswell,et al.  The developing constraints on parsing decisions: The role of lexical-biases and referential scenes in child and adult sentence processing , 2004, Cognitive Psychology.

[66]  Julie C. Sedivy,et al.  Subject Terms: Linguistics Language Eyes & eyesight Cognition & reasoning , 1995 .

[67]  Daniel C. Richardson,et al.  Exploring the movement dynamics of deception , 2013, Front. Psychol..

[68]  D. Kemmerer Are the motor features of verb meanings represented in the precentral motor cortices? Yes, but within the context of a flexible, multilevel architecture for conceptual knowledge , 2015, Psychonomic Bulletin & Review.

[69]  G. Rizzolatti,et al.  Listening to action-related sentences modulates the activity of the motor system: a combined TMS and behavioral study. , 2005, Brain research. Cognitive brain research.

[70]  D. Pisoni,et al.  Reaction times to comparisons within and across phonetic categories , 1974, Perception & psychophysics.

[71]  Rolf A. Zwaan Embodiment and language comprehension: reframing the discussion , 2014, Trends in Cognitive Sciences.

[72]  D. Norris,et al.  Shortlist B: a Bayesian model of continuous speech recognition. , 2008, Psychological review.

[73]  Caden Salvata,et al.  Temporal Cortex Reflects Effects of Sentence Context on Phonetic Processing , 2013, Journal of Cognitive Neuroscience.

[74]  Friedemann Pulvermüller,et al.  Motor cortex maps articulatory features of speech sounds , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[75]  Rolf A. Zwaan,et al.  Seeing, acting, understanding: motor resonance in language comprehension. , 2006, Journal of experimental psychology. General.

[76]  T. Dijkstra,et al.  Making sense: motor activation and action plausibility during sentence processing , 2017 .

[77]  Daniel Casasanto,et al.  Flexibility in Embodied Language Understanding , 2011, Front. Psychology.

[78]  Alexandra Paxton,et al.  Argument disrupts interpersonal synchrony , 2013, Quarterly journal of experimental psychology.

[79]  Kara D. Federmeier,et al.  Alpha and theta band dynamics related to sentential constraint and word expectancy , 2017, Language, cognition and neuroscience.

[80]  Michael J. Spivey,et al.  Unraveling the Dyad: Using Recurrence Analysis to Explore Patterns of Syntactic Coordination Between Children and Caregivers in Conversation , 2006 .

[81]  James L. McClelland,et al.  The TRACE model of speech perception , 1986, Cognitive Psychology.

[82]  Daniel C. Richardson,et al.  Spatial representations activated during real-time comprehension of verbs , 2003, Cogn. Sci..

[83]  Michael P. Kaschak,et al.  Grounding language in action , 2002, Psychonomic bulletin & review.

[84]  Mark Steedman,et al.  Interaction with context during human sentence processing , 1988, Cognition.

[85]  G. V. van Orden,et al.  Self-organization of cognitive performance. , 2003, Journal of experimental psychology. General.

[86]  Daniel C. Richardson,et al.  The self organization of human interaction , 2013 .

[87]  Christopher M. Crew,et al.  The Cambridge Handbook of Psycholinguistics: Research in Language Production , 2012 .

[88]  Paul Smolensky,et al.  Harmony in Linguistic Cognition , 2006, Cogn. Sci..

[89]  B. McElree,et al.  Syntactic and Thematic Processing in Sentence Comprehension: Evidence for a Temporal Dissociation , 1995 .

[90]  W. Ganong Phonetic categorization in auditory word perception. , 1980, Journal of experimental psychology. Human perception and performance.

[91]  Steven H. Strogatz,et al.  Sync: The Emerging Science of Spontaneous Order , 2003 .

[92]  Sarah Brown-Schmidt,et al.  The rapid use of gender information: evidence of the time course of pronoun resolution from eyetracking , 2000, Cognition.

[93]  Julie C. Sedivy,et al.  Eye movements to pictures reveal transient semantic activation during spoken word recognition. , 2006, Journal of experimental psychology. Learning, memory, and cognition.

[94]  Jennifer S. Pardo,et al.  Measuring phonetic convergence in speech production , 2013, Front. Psychol..

[95]  A. Opstal Dynamic Patterns: The Self-Organization of Brain and Behavior , 1995 .

[96]  Marco Iacoboni,et al.  The role of premotor cortex in speech perception: Evidence from fMRI and rTMS , 2008, Journal of Physiology-Paris.

[97]  Adrian Staub,et al.  Word recognition and syntactic attachment in reading: evidence for a staged architecture. , 2011, Journal of experimental psychology. General.

[98]  Riccardo Fusaroli,et al.  Investigating Conversational Dynamics: Interactive Alignment, Interpersonal Synergy, and Collective Task Performance , 2016, Cogn. Sci..

[99]  Marta Kutas,et al.  The phonemic restoration effect reveals pre-N400 effect of supportive sentence context in speech perception , 2010, Brain Research.

[100]  D. Tranel,et al.  A DOUBLE DISSOCIATION BETWEEN LINGUISTIC AND PERCEPTUAL REPRESENTATIONS OF SPATIAL RELATIONSHIPS , 2000, Cognitive neuropsychology.

[101]  Daniel C. Richardson,et al.  Motor representations in memory and mental models: Embodiment in cognition , 2001 .

[102]  Mark Steedman,et al.  Ambiguity in context: A reply , 1989 .

[103]  Stephen Grossberg,et al.  Adaptive Resonance Theory: How a brain learns to consciously attend, learn, and recognize a changing world , 2013, Neural Networks.

[104]  Jon Andoni Duñabeitia,et al.  Brain-to-brain entrainment: EEG interbrain synchronization while speaking and listening , 2017, Scientific Reports.

[105]  Jeff Yoshimi,et al.  Active internalism and open dynamical systems , 2012 .

[106]  Alex Clarke,et al.  Brain oscillations in language comprehension , 2017 .