Contrast Is in the Eye of the Beholder: Infelicitous Beat Gesture Increases Cognitive Load During Online Spoken Discourse Comprehension

We investigated how two cues to contrast-beat gesture and contrastive pitch accenting-affect comprehenders' cognitive load during processing of spoken referring expressions. In two visual-world experiments, we orthogonally manipulated the presence of these cues and their felicity, or fit, with the local (sentence-level) referential context in critical referring expressions while comprehenders' task-evoked pupillary responses (TEPRs) were examined. In Experiment 1, beat gesture and contrastive accenting always matched the referential context of filler referring expressions and were therefore relatively felicitous on the global (experiment) level, whereas in Experiment 2, beat gesture and contrastive accenting never fit the referential context of filler referring expressions and were therefore infelicitous on the global level. The results revealed that both beat gesture and contrastive accenting increased comprehenders' cognitive load. For beat gesture, this increase in cognitive load was driven by both local and global infelicity. For contrastive accenting, this increase in cognitive load was unaffected when cues were globally felicitous but exacerbated when cues were globally infelicitous. Together, these results suggest that comprehenders' cognitive resources are taxed by processing infelicitous use of beat gesture and contrastive accenting to convey contrast on both the local and global levels.

[1]  Mickaël Causse,et al.  High Working Memory Load Impairs Language Processing during a Simulated Piloting Task: An ERP and Pupillometry Study , 2016, Front. Hum. Neurosci..

[2]  A. Wingfield,et al.  Pupillometry as a measure of cognitive effort in younger and older adults. , 2010, Psychophysiology.

[3]  K. Demuth,et al.  Non-referential gestures in adult and child speech: Are they prosodic? , 2016 .

[4]  J. Beatty.  Task-evoked pupillary responses, processing load, and the structure of processing resources. , 1982, Psychological bulletin.

[5]  D. Leutner,et al.  Assessment of Cognitive Load in Multimedia Learning with Dual-Task Methodology: Auditory Load and Modality Effects , 2004 .

[6]  Stefan Frank,et al.  Early effects of word surprisal on pupil size during reading , 2012, CogSci.

[7]  E. Hess,et al.  Pupil Size in Relation to Mental Activity during Simple Problem-Solving , 1964, Science.

[8]  P. Ferré,et al.  Is pupillary response a reliable index of word recognition? Evidence from a delayed lexical decision task , 2017, Behavior research methods.

[9]  Megan H. Papesh,et al.  Pupil-BLAH-metry: Cognitive effort in speech planning reflected by pupil dilation , 2012, Attention, perception & psychophysics.

[10]  Benjamin Naumann,et al.  Mental Representations A Dual Coding Approach , 2016 .

[11]  Susan Goldin-Meadow,et al.  Gesturing Saves Cognitive Resources When Talking About Nonpresent Objects , 2010, Cogn. Sci..

[12]  P. Ferré,et al.  Pupil dilation is sensitive to the cognate status of words: further evidence for non-selectivity in bilingual lexical access , 2017 .

[13]  P. Toffanin,et al.  The Timing and Effort of Lexical Access in Natural and Degraded Speech , 2016, Front. Psychol..

[14]  R. Klatzky,et al.  Teachers’ gestures facilitate students’ learning: A lesson in symmetry , 2003 .

[15]  Julie C. Sedivy,et al.  Subject Terms: Linguistics Language Eyes & eyesight Cognition & reasoning , 1995 .

[16]  James C. Lester,et al.  The Case for Social Agency in Computer-Based Teaching: Do Students Learn More Deeply When They Interact With Animated Pedagogical Agents? , 2001 .

[17]  Wing Chee So,et al.  Mnemonic effect of iconic gesture and beat gesture in adults and children: Is meaning in gesture important for memory recall? , 2012 .

[18]  Mari Ostendorf,et al.  TOBI: a standard for labeling English prosody , 1992, ICSLP.

[19]  Mariska E. Kret,et al.  Preprocessing pupil size data: Guidelines and code , 2018, Behavior Research Methods.

[20]  D. McNeill.  Hand and Mind , 1995 .

[21]  P. Prieto,et al.  The Effects of pitch accentuation and beat gestures on information recall in contrastive discourse , 2016 .

[22]  A. Jacobs,et al.  Pupillary responses during lexical decisions vary with word frequency but not emotional valence. , 2007, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[23]  Núria Esteve-Gibert,et al.  Prosodic structure shapes the temporal realization of intonation and manual gesture movements. , 2013, Journal of speech, language, and hearing research : JSLHR.

[24]  Laura M. Morett,et al.  Emphasising sound and meaning: pitch gestures enhance Mandarin lexical tone acquisition , 2015 .

[25]  Daniel Mirman,et al.  GazeR: A Package for Processing Gaze Position and Pupil Size Data , 2020, Behavior research methods.

[26]  J. Sweller,et al.  Reducing cognitive load by mixing auditory and visual presentation modes , 1995 .

[27]  Sebastian Sauppe.  Symmetrical and asymmetrical voice systems and processing load: Pupillometric evidence from sentence production in Tagalog and German , 2017 .

[28]  Julie C. Sedivy,et al.  Achieving incremental semantic interpretation through contextual representation , 1999, Cognition.

[29]  Pilar Prieto,et al.  Beat gestures help preschoolers recall and comprehend discourse information. , 2018, Journal of experimental child psychology.

[30]  Stefanie E. Kuchinsky,et al.  Pupil size varies with word listening and response selection difficulty in older adults with hearing loss. , 2013, Psychophysiology.

[31]  John Myhill,et al.  Towards an operational definition of discourse contrast , 1996 .

[32]  Michael J. Crosse,et al.  Congruent Visual Speech Enhances Cortical Entrainment to Continuous Auditory Speech in Noise-Free Conditions , 2015, The Journal of Neuroscience.

[33]  D Kahneman,et al.  Pupil Diameter and Load on Memory , 1966, Science.

[34]  Per B. Brockhoff,et al.  lmerTest Package: Tests in Linear Mixed Effects Models , 2017 .

[35]  C. Clifton,et al.  The independence of syntactic processing , 1986 .

[36]  Elisabeth Selkirk,et al.  Sentence Prosody: Intonation, Stress and Phrasing , 1996 .

[37]  Jason Geller,et al.  Eyes wide open: Pupil size as a proxy for inhibition in the masked-priming paradigm , 2015, Memory & Cognition.

[38]  Laura C. Dilley,et al.  role of f0 alignment in distinguishing intonation categories: evidence from American english , 2021 .

[39]  Ellen Campana,et al.  Speech-and-gesture integration in high functioning autism , 2010, Cognition.

[40]  Elena G. Patsenko,et al.  The Quarterly Journal of Experimental Psychology Pupillometry Reveals Processing Load during Spoken Language Comprehension , 2022 .

[41]  D. House.  Tonal perception in speech , 1990 .

[42]  J. Masefield,et al.  "Sagging transitions" between high pitch accents in English: experimental evidence , 2003, J. Phonetics.

[43]  Naomi Sweller,et al.  Presentation and production: the role of gesture in spatial communication. , 2014, Journal of experimental child psychology.

[44]  D. Barr,et al.  Random effects structure for confirmatory hypothesis testing: Keep it maximal. , 2013, Journal of memory and language.

[45]  S. Sirois,et al.  Pupillometry , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[46]  Michael K. Tanenhaus,et al.  Interpreting Pitch Accents in Online Comprehension: H* vs. L+H , 2008, Cogn. Sci..

[47]  R. Krauss,et al.  Word Familiarity Predicts Temporal Asynchrony of Hand Gestures and Speech , 2010 .

[48]  Anne Lazaraton,et al.  Gesture and Speech in the Vocabulary Explanations of One ESL Teacher: A Microanalytic Inquiry. , 2004 .

[49]  Pilar Prieto i Vives,et al.  Deconstructing beat gestures: a labelling proposal , 2018, Speech Prosody 2018.

[50]  Jacolien van Rij,et al.  Individual Differences in Pupil Dilation during Naming Task , 2016, CogSci.

[51]  Barbara G. Shinn-Cunningham,et al.  The pupil response reveals increased listening effort when it is difficult to focus attention , 2015, Hearing Research.

[52]  Heather Leavy Rusiewicz,et al.  Synchronization of Prosodic Stress and Gesture : A Dynamic Systems Perspective , 2011 .

[53]  M. Swerts,et al.  The Effects of Visual Beats on Prosodic Prominence: Acoustic Analyses, Auditory Perception and Visual Perception. , 2007 .

[54]  Michael Gasser,et al.  A Dynamic Approach to Rhythm in Language: Toward a Temporal Phonology , 1995, ArXiv.

[55]  Matthew B. Winn,et al.  The Impact of Auditory Spectral Resolution on Listening Effort Revealed by Pupil Dilation , 2015, Ear and hearing.

[56]  Esther Janse,et al.  Speech rate effects on the processing of conversational speech across the adult life span. , 2016, The Journal of the Acoustical Society of America.

[57]  E. Vatikiotis-Bateson,et al.  Perceiving Biological Motion: Dissociating Visible Speech from Walking , 2003, Journal of Cognitive Neuroscience.

[58]  Edward Gibson,et al.  Inter-transcriber reliability for two systems of prosodic annotation: ToBI (Tones and Break Indices) and RaP (Rhythm and Pitch) , 2012 .

[59]  P. Prieto,et al.  Prominence in speech and gesture favour second language novel word learning , 2018 .

[60]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[61]  Torsten Dau,et al.  Impact of Background Noise and Sentence Complexity on Processing Demands during Sentence Comprehension , 2016, Front. Psychol..

[62]  Laura C. Dilley,et al.  The RaP (Rhythm and Pitch) Labeling System , 2005 .

[63]  Peter Hagoort,et al.  The Quarterly Journal of Experimental Psychology Pupillometry Reveals Increased Pupil Size during Indirect Request Comprehension , 2022 .

[64]  Stefanie Shattuck-Hufnagel,et al.  THE TIMING OF SPEECH-ACCOMPANYING GESTURES WITH RESPECT TO PROSODY , 2004 .

[65]  Vera Demberg,et al.  The Frequency of Rapid Pupil Dilations as a Measure of Linguistic Processing Difficulty , 2016, PloS one.

[66]  Rachel Ryskin,et al.  Information Integration in Modulation of Pragmatic Inferences During Online Language Comprehension , 2019, Cogn. Sci..

[67]  Michael Schluroff.  Pupil responses to grammatical complexity of sentences , 1982, Brain and Language.

[68]  D. Bolinger.  A Theory of Pitch Accent in English , 1958 .

[69]  Sotaro Kita,et al.  Conceptualisation load triggers gesture production , 2007 .

[70]  Kiwako Ito,et al.  Anticipatory effects of intonation: Eye movements during instructed visual search. , 2008, Journal of memory and language.

[71]  Anne-Lise Giraud,et al.  Prediction across sensory modalities: A neurocomputational model of the McGurk effect , 2015, Cortex.

[72]  Kara D. Federmeier,et al.  Verbal working memory predicts co-speech gesture: Evidence from individual differences , 2014, Cognition.

[73]  Joshua F. Wiley,et al.  Growth Curve Analysis and Visualization Using R , 2014 .

[74]  D. Mirman,et al.  A Pupillometric Examination of Cognitive Control in Taxonomic and Thematic Semantic Memory , 2019, Journal of cognition.

[75]  Susan Shaiman,et al.  Effects of perturbation and prosody on the coordination of speech and gesture , 2014, Speech Commun..

[76]  S. Goldin-Meadow,et al.  Hearing Gesture: How Our Hands Help Us Think , 2003 .

[77]  Jeffery A. Jones,et al.  Neural processes underlying perceptual enhancement by visual speech gestures , 2003, Neuroreport.

[78]  Fred Cummins,et al.  The temporal relation between beat gestures and speech , 2011 .

[79]  Steven G. McCafferty,et al.  Gesture and Creating Zones of Proximal Development for Second Language Learning , 2002 .

[80]  Jamie Reilly,et al.  The human task-evoked pupillary response function is linear: Implications for baseline response scaling in pupillometry , 2018, Behavior Research Methods.

[81]  Anna Piotrowska,et al.  Processing load during listening: The influence of task characteristics on the pupil response , 2013 .

[82]  Y. Ben-nun.  The use of pupillometry in the study of on-line verbal processing: Evidence for depths of processing , 1986, Brain and Language.

[83]  Susan M. Wagner,et al.  Explaining Math: Gesturing Lightens the Load , 2001, Psychological science.

[84]  Stefanie E. Kuchinsky,et al.  Pupillometry reveals changes in physiological arousal during a sustained listening task. , 2017, Psychophysiology.

[85]  M. Just,et al.  From the SelectedWorks of Marcel Adam Just 1992 A capacity theory of comprehension : Individual differences in working memory , 2017 .

[86]  Sidney K. D'Mello,et al.  Gesturing May Not Always Make Learning Last , 2014, CogSci.

[87]  P. Prieto,et al.  Observing Storytellers Who Use Rhythmic Beat Gestures Improves Children’s Narrative Discourse Performance , 2019, Developmental psychology.

[88]  D. McNeill.  Gesture and Thought , 2005 .

[89]  S. Mathôt.  Pupillometry: Psychology, Physiology, and Function , 2018, Journal of cognition.

[90]  Elisabeth Selkirk,et al.  Contrastive FOCUS vs. presentational focus: prosodic evidence from right node raising in English , 2002, Speech Prosody 2002.

[91]  M. H. Fischer,et al.  Listening to Limericks: A Pupillometry Investigation of Perceivers’ Expectancy , 2013, PloS one.

[92]  Esteban Buz,et al.  The concurrent use of three implicit measures (eye movements, pupillometry, and event-related potentials) to assess receptive vocabulary knowledge in normal adults , 2016, Behavior research methods.

[93]  Mora Maldonado,et al.  Pupillary Dynamics Reveal Computational Cost in Sentence Planning , 2014, Quarterly journal of experimental psychology.

[94]  Laura C. Dilley,et al.  Pitch Range Variation in English Tonal Contrasts: Continuous or Categorical? , 2010, Phonetica.

[95]  Martina Zellin,et al.  In the eye of the listener: pupil dilation elucidates discourse processing. , 2011, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[96]  Susan M. Wagner,et al.  Probing the Mental Representation of Gesture: Is Handwaving Spatial?. , 2004 .

[97]  Laura M. Morett,et al.  Listeners consider alternative speaker productions in discourse comprehension and memory: Evidence from beat gesture and pitch accenting , 2019, Memory & Cognition.

[98]  Susan Shaiman,et al.  Effects of prosody and position on the timing of deictic gestures. , 2013, Journal of speech, language, and hearing research : JSLHR.

[99]  Pilar Prieto,et al.  Beat gestures improve word recall in 3- to 5-year-old children. , 2017, Journal of experimental child psychology.

[100]  Barbara G. Shinn-Cunningham,et al.  The pupil response is sensitive to divided attention during speech processing , 2014, Hearing Research.

[101]  Duane G. Watson,et al.  Recognition memory reveals just how CONTRASTIVE contrastive accenting really is. , 2010, Journal of memory and language.

[102]  Daichi,et al.  Effects of Semantic Information and Punctuation in Processing Japanese Garden-Path Sentences: Evidence from Pupillary Responses , 2016 .

[103]  M. Just,et al.  The intensity dimension of thought: pupillometric indices of sentence processing. , 1993, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[104]  Julie C. Sedivy,et al.  The effect of speaker-specific information on pragmatic inferences , 2011 .

[105]  D. Leutner,et al.  Direct Measurement of Cognitive Load in Multimedia Learning , 2003 .

[106]  Stefanie Shattuck-Hufnagel,et al.  Tonal Center of Gravity: A global approach to tonal implementation in a level-based intonational phonology , 2012 .

[107]  Emiel Krahmer,et al.  Perceptual evaluation of audiovisual cues for prominence , 2002, INTERSPEECH.

[108]  David House,et al.  Acoustic features of multimodal prominences: Do visual beat gestures affect verbal pitch accent realization? , 2017, AVSP.

[109]  Lars Marstaller,et al.  Individual differences in the gesture effect on working memory , 2013, Psychonomic bulletin & review.

[110]  Mitchell J. Nathan,et al.  Teachers’ gestures and students’ learning: sometimes “hands off” is better , 2017, Cognitive Research: Principles and Implications.

[111]  Barbara Höhle,et al.  Pupillometry registers toddlers’ sensitivity to degrees of mispronunciation , 2017, Journal of experimental child psychology.

[112]  Daniel F. Pontillo,et al.  Is it or isn’t it: Listeners make rapid use of prosody to infer speaker meanings , 2014, Cognition.

[113]  Jens Schmidtke.  Second language experience modulates word retrieval effort in bilinguals: evidence from pupillometry , 2014, Front. Psychol..

[114]  R. Mayer,et al.  Nine Ways to Reduce Cognitive Load in Multimedia Learning , 2003 .

[115]  P. Pexman,et al.  Getting a grip on sensorimotor effects in lexical–semantic processing , 2018, Behavior research methods.

[116]  J Hyönä,et al.  Pupil Dilation as a Measure of Processing Load in Simultaneous Interpretation and Other Language Tasks , 1995, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[117]  A. Zekveld,et al.  Pupil Response as an Indication of Effortful Listening: The Influence of Sentence Intelligibility , 2010, Ear and hearing.

[118]  L. R. Chapman,et al.  A Novel Pupillometric Method for Indexing Word Difficulty in Individuals With and Without Aphasia. , 2015, Journal of speech, language, and hearing research : JSLHR.

[119]  Duane G. Watson.  Chapter 4 - The Many Roads to Prominence: Understanding Emphasis in Conversation , 2010 .

[120]  Ken McRae,et al.  Constraint-based models of sentence processing , 2013 .

[121]  D. Leutner,et al.  Assessment of cognitive load in multimedia learning using dual-task methodology. , 2002, Experimental psychology.