Activating Semantic Knowledge During Spoken Words and Environmental Sounds: Evidence From the Visual World Paradigm

Two visual world experiments investigated the activation of semantically related concepts during the processing of environmental sounds and spoken words. Participants heard environmental sounds such as barking or spoken words such as "puppy" while viewing visual arrays with objects such as a bone (semantically related competitor) and candle (unrelated distractor). In Experiment 1, a puppy (target) was also included in the visual array; in Experiment 2, it was not. During both types of auditory stimuli, competitors were fixated significantly more than distractors, supporting the coactivation of semantically related concepts in both cases; comparisons of the two types of auditory stimuli also revealed significantly larger effects with environmental sounds than spoken words. We discuss implications of these results for theories of semantic knowledge.

[1]  Guillaume Thierry,et al.  Hemispheric Dissociation in Access to the Human Semantic System , 2003, Neuron.

[2]  Allan Collins,et al.  A spreading-activation theory of semantic processing , 1975 .

[3]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[4]  Charles Spence,et al.  Crossmodal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity. , 2011, Journal of experimental psychology. Human perception and performance.

[5]  C. Petten,et al.  Conceptual relationships between spoken words and environmental sounds: Event-related brain potential measures , 1995, Neuropsychologia.

[6]  F. Dick,et al.  An on-line task for contrasting auditory processing in the verbal and nonverbal domains and norms for younger and older adults , 2005, Behavior research methods.

[7]  Falk Huettig,et al.  A stimulus set of words and pictures matched for visual and semantic similarity , 2016 .

[8]  James L. McClelland,et al.  The parallel distributed processing approach to semantic cognition , 2003, Nature Reviews Neuroscience.

[9]  Yuki Kamide,et al.  Spatial narrative context modulates semantic (but not visual) competition during discourse processing , 2019, Journal of Memory and Language.

[10]  C. Spence,et al.  Audiovisual Semantic Interactions Between Linguistic and Nonlinguistic Stimuli: The Time-Courses and Categorical Specificity , 2018, Journal of experimental psychology. Human perception and performance.

[11]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[12]  Brian Gygi,et al.  Individual differences in auditory abilities. , 2007, The Journal of the Acoustical Society of America.

[13]  Effects of Context on the Identification of Everyday Sounds , 1991 .

[14]  Lawrence W. Barsalou,et al.  Perceptions of perceptual symbols , 1999, Behavioral and Brain Sciences.

[15]  Siddharth Singh,et al.  Activation of shape and semantic information during ambiguous homophone processing: eye tracking evidence from Hindi , 2014, Cognitive Processing.

[16]  Andreas Mädebach,et al.  Neighing, Barking, and Drumming Horses—Object Related Sounds Help and Hinder Picture Naming , 2017, Journal of experimental psychology. Human perception and performance.

[17]  Dissociating the time courses of the cross-modal semantic priming effects elicited by naturalistic sounds and spoken words , 2017, Psychonomic bulletin & review.

[18]  S. Thompson-Schill,et al.  Putting concepts into context , 2016, Psychonomic bulletin & review.

[19]  M. Grabowecky,et al.  Characteristic sounds facilitate visual search , 2008, Psychonomic bulletin & review.

[20]  F. Dick,et al.  Infants' Recognition of Meaningful Verbal and Nonverbal Sounds , 2009, Language learning and development : the official journal of the Society for Language Development.

[21]  M. Grabowecky,et al.  Characteristic sounds make you look at target objects more quickly , 2010, Attention, perception & psychophysics.

[22]  Tracy Love,et al.  The organization of words and environmental sounds in memory , 2015, Neuropsychologia.

[23]  S. McDonald,et al.  Models of high-dimensional semantic space predict language-mediated eye movements in the visual world. , 2006, Acta psychologica.

[24]  Ayse Pinar Saygin,et al.  What is Involved and What is Necessary for Complex Linguistic and Nonlinguistic Auditory Processing: Evidence from Functional Magnetic Resonance Imaging and Lesion Data , 2007, Journal of Cognitive Neuroscience.

[25]  Roger M. Cooper,et al.  The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. , 1974 .

[26]  F. Dick,et al.  Neural resources for processing language and environmental sounds: evidence from aphasia. , 2003, Brain : a journal of neurology.

[27]  S. Thompson-Schill,et al.  The evocative power of words: activation of concepts by verbal and nonverbal means. , 2012, Journal of experimental psychology. General.

[28]  Sharon L. Thompson-Schill,et al.  Looking for meaning: Eye movements are sensitive to overlapping semantic features, not association , 2009, Psychonomic bulletin & review.

[29]  Falk Huettig,et al.  When meaning matters: The temporal dynamics of semantic influences on visual attention. , 2016, Journal of experimental psychology. Human perception and performance.

[30]  G. Lupyan,et al.  What makes words special? Words as unmotivated cues , 2015, Cognition.

[31]  Marc Brysbaert,et al.  Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English , 2009, Behavior research methods.

[32]  Curt Burgess,et al.  Producing high-dimensional semantic spaces from lexical co-occurrence , 1996 .

[33]  D. Mirman,et al.  Statistical and computational models of the visual world paradigm: Growth curves and individual differences. , 2008, Journal of memory and language.

[34]  D. Mirman,et al.  Dynamics of activation of semantically similar concepts during spoken word recognition , 2009, Memory & cognition.

[35]  G. Altmann,et al.  Word meaning and the control of eye fixation: semantic competitor effects and the visual world paradigm , 2005, Cognition.

[36]  Julie C. Sedivy,et al.  Subject Terms: Linguistics Language Eyes & eyesight Cognition & reasoning , 1995 .

[37]  Julie C. Sedivy,et al.  Eye movements to pictures reveal transient semantic activation during spoken word recognition. , 2006, Journal of experimental psychology. Learning, memory, and cognition.

[38]  D. Mirman,et al.  Individual differences in the strength of taxonomic versus thematic relations. , 2012, Journal of experimental psychology. General.

[39]  Marcia Grabowecky,et al.  Object-based auditory facilitation of visual search for pictures and words with frequent and rare targets. , 2011, Acta psychologica.

[40]  Guillaume Thierry,et al.  Dissociating Verbal and Nonverbal Conceptual Processing in the Human Brain , 2006, Journal of Cognitive Neuroscience.

[41]  Julie C. Sedivy,et al.  Playing on the typewriter, typing on the piano: manipulation knowledge of objects , 2006, Cognition.