Cognitive Load Increases Spoken and Gestural Hesitation Frequency

This study investigates the interplay of spoken and gestural hesitations under varying amounts of cognitive load. We argue that not only fillers and silences, as the most common hesitations, are directly related to speech pausing behavior, but that hesitation lengthening is as well. We designed a resource-management card game as a method to elicit ecologically valid pausing behavior while being able to finely control cognitive load via card complexity. The method very successfully elicits large amounts of hesitations. Hesitation frequency increases as a function of cognitive load. This is true for both spoken and gestural hesitations. We conclude that the method presented here is a versatile tool for future research and we present foundational research on the speech-gesture link related to hesitations induced by controllable cognitive load.

[1]  Farhat Jabeen,et al.  Hesitations in Urdu/Hindi: Distribution and Properties of Fillers & Silences , 2022, INTERSPEECH.

[2]  C. Stepp,et al.  Changes in Relative Fundamental Frequency Under Increased Cognitive Load in Individuals With Healthy Voices. , 2021, Journal of speech, language, and hearing research : JSLHR.

[3]  P. Marentette,et al.  How Referential Gestures Align With Speech: Evidence From Monolingual and Bilingual Speakers , 2020, Language Learning.

[4]  John Thangarajah,et al.  Estimating cognitive load from speech gathered in a complex real-life training exercise , 2019, Int. J. Hum. Comput. Stud..

[5]  James A. Dixon,et al.  Entrainment and Modulation of Gesture–Speech Synchrony Under Delayed Auditory Feedback , 2018, Cogn. Sci..

[6]  Marianne Gullberg,et al.  When Speech Stops, Gesture Stops: Evidence From Developmental and Crosslinguistic Comparisons , 2018, Front. Psychol..

[7]  Petra Wagner,et al.  Interactive Hesitation Synthesis: Modelling and Evaluation , 2018 .

[8]  Per B. Brockhoff,et al.  lmerTest Package: Tests in Linear Mixed Effects Models , 2017 .

[9]  Florian Schiel,et al.  Multilingual processing of speech via web services , 2017, Comput. Speech Lang..

[10]  S. Goldin-Meadow,et al.  Gesture as representational action: A paper about function , 2016, Psychonomic Bulletin & Review.

[11]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[12]  Stefan Kopp,et al.  Gesture and speech in interaction: An overview , 2014, Speech Commun..

[13]  S. Goldin-Meadow,et al.  Gesturing makes learning last , 2008, Cognition.

[14]  Jennifer E. Arnold,et al.  If you say thee uh you are describing something hard: the on-line attribution of disfluency during reference comprehension. , 2007, Journal of experimental psychology. Learning, memory, and cognition.

[15]  Susan M. Wagner,et al.  Explaining Math: Gesturing Lightens the Load , 2001, Psychological science.

[16]  J. D. Ruiter.  The production of gesture and speech , 2000 .

[17]  Sotaro Kita,et al.  How representational gestures help speaking , 2000 .

[18]  S. Goldin-Meadow,et al.  The role of gesture in communication and thinking , 1999, Trends in Cognitive Sciences.

[19]  J. E. Tree.  The Effects of False Starts and Repetitions on the Processing of Subsequent Words in Spontaneous Speech , 1995 .

[20]  D B Pisoni,et al.  Effects of cognitive workload on speech production: acoustic analyses and perceptual consequences. , 1993, The Journal of the Acoustical Society of America.

[21]  P. Chandler,et al.  Evidence for Cognitive Load Theory , 1991 .

[22]  Anne H. Anderson,et al.  The Hcrc Map Task Corpus , 1991 .

[23]  Joakim Nivre,et al.  Speech Management—on the Non-written Life of Speech , 1990, Nordic Journal of Linguistics.

[24]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[25]  F. Goldman-Eisler,et al.  Sequential Temporal Patterns and Cognitive Processes in Speech , 1967, Language and speech.

[26]  Malte Belz.  Die Phonetik von äh und ähm , 2021 .

[27]  J. Varma.  Interactive Gestures , 2019, SwiftUI for Absolute Beginners.

[28]  Petra Wagner,et al.  In defense of stylistic diversity in speech research , 2015, J. Phonetics.

[29]  Claude Montacié,et al.  High-level speech event analysis for cognitive load classification , 2014, INTERSPEECH.

[30]  Fabien Ringeval,et al.  The INTERSPEECH 2014 computational paralinguistics challenge: cognitive & physical load , 2014, INTERSPEECH.

[31]  Eliathamby Ambikairajah,et al.  Formant Frequencies under Cognitive Load: Effects and Classification , 2011, EURASIP J. Adv. Signal Process..

[32]  R. Krauss,et al.  Word Familiarity Predicts Temporal Asynchrony of Hand Gestures and Speech , 2010 .

[33]  Zofia Malisz,et al.  Aspects of gestural and prosodic structure of multimodal utterances in Polish task-oriented dialogues , 2008 .

[34]  Julie A. Jacko,et al.  Human-Computer Interaction. Interaction Design and Usability, 12th International Conference, HCI International 2007, Beijing, China, July 22-27, 2007, Proceedings, Part I , 2007, HCI.

[35]  D. McNeill.  Gesture and Thought , 2005 .

[36]  Sotaro Kita,et al.  What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking , 2003 .

[37]  G. Beattie,et al.  Cross-cultural similarities in gestures: The deep relationship between gestures and speech which transcends language barriers , 1996 .

[38]  Robin N. Campbell,et al.  Recent Advances in the Psychology of Language , 1978 .