Cross-modal mapping of visual and acoustic displays in infants: The effect of dynamic and static components

Children as well as adults make certain links between the visual and the acoustic domain. For example, higher pitched sounds are typically associated with visual stimuli that are brighter, smaller, and higher in space than those associated with sounds of lower pitch. The present study explored the existence of such links in infancy. More specifically, we investigated the relation between visuospatial height and auditory pitch in 7- to 12-month-olds, varying systematically the format of stimulus presentation. Presenting dynamic visual and acoustic stimuli led to positive findings relying on anticipatory looking measures: While listening to an ascending tone, infants looked longer where they had previously seen an upwards moving circle. While listening to a descending tone, they looked longer where they had previously seen a downwards moving circle. When static tones were combined with static visual stimuli in the same kind of display, no corresponding findings were obtained. Thus, the present work suggests that preverbal infants do not generally associate spatial location and pitch but do so systematically when dynamic stimuli are presented.

[1]  J. Simner.  Defining synaesthesia. , 2012, British journal of psychology.

[2]  C. C. Pratt.  The spatial character of high and low tones. , 1930 .

[3]  L E Marks,et al.  On associations of light and sound: the mediation of brightness, pitch, and loudness. , 1974, The American journal of psychology.

[4]  Marcia Grabowecky,et al.  Changes in auditory frequency guide visual–spatial attention , 2011, Cognition.

[5]  J. Morais,et al.  Spatial associations for musical stimuli: a piano in the head? , 2007, Journal of experimental psychology. Human perception and performance.

[6]  A. Walker,et al.  Intermodal perception of expressive behaviors by human infants. , 1982, Journal of experimental child psychology.

[7]  J. Simner,et al.  Early detection of markers for synaesthesia in childhood populations. , 2009, Brain : a journal of neurology.

[8]  A. Walker-Andrews.  Infants' perception of the affordances of expressive behaviors. , 1988 .

[9]  Lawrence E. Marks,et al.  Visual-auditory interaction in speeded classification: Role of stimulus difference , 1995, Perception & psychophysics.

[10]  L. Bahrick.  Infants' perceptual differentiation of amodal and modality-specific audio-visual relations. , 1992, Journal of experimental child psychology.

[11]  B. Morrongiello,et al.  Sound localization in newborn human infants. , 1994, Developmental psychobiology.

[12]  E S Spelke,et al.  The infant's acquisition of knowledge of bimodally specified events. , 1981, Journal of experimental child psychology.

[13]  E. Spelke,et al.  Numerical abstraction by human infants , 1990, Cognition.

[14]  Ryota Kanai,et al.  Changing pitch induced visual motion illusion , 2004, Current Biology.

[15]  J Field,et al.  Newborn infants orient to sounds. , 1979, Child development.

[16]  L. Marks,et al.  Processes underlying dimensional interactions: Correspondences between linguistic and nonlinguistic dimensions , 1990, Memory & cognition.

[17]  Lawrence E. Marks,et al.  On colored-hearing synesthesia: cross-modal translations of sensory dimensions. , 1975 .

[18]  L. Marks.  The Unity of the Senses: Interrelations Among the Modalities , 1978 .

[19]  Bruno L. Giordano,et al.  Spatial representation of pitch height: the SMARC effect , 2006, Cognition.

[20]  Barbara Dodd,et al.  Lip reading in infants: Attention to speech presented in- and out-of-synchrony , 1979, Cognitive Psychology.

[21]  E. Spelke.  Perceiving Bimodally Specified Events in Infancy , 1979 .

[22]  A. Walker-Andrews.  Intermodal perception of expressive behaviors: Relation of eye and voice? , 1986 .

[23]  R K Clifton,et al.  Newborns' orientation toward sound: possible implications for cortical development. , 1981, Child development.

[24]  Renee Timmers,et al.  Beethoven’s last piano sonata and those who follow crocodiles: Cross-domain mappings of auditory pitch in a musical context , 2010, Cognition.

[25]  R. Butler,et al.  Localization of tonal stimuli in the vertical plane. , 1968, The Journal of the Acoustical Society of America.

[26]  D. Maurer,et al.  Synesthesia: a new approach to understanding the development of perception. , 2013, Developmental psychology.

[27]  W. Abraham,et al.  The ontogenesis of an auditory localization response from birth to four months of age. , 1979, Canadian journal of psychology.

[28]  D. Maurer.  Neonatal Synesthesia: Implications for the Processing of Speech And Faces , 1993 .

[29]  L. Bahrick,et al.  Infants' Bimodal Perception of Gender , 1991 .

[30]  L E Marks,et al.  Perceiving similarity and comprehending metaphor. , 1988, Monographs of the Society for Research in Child Development.

[31]  Z. Eitan,et al.  HOW MUSIC MOVES: Musical Parameters and Listeners' Images of Motion , 2006 .

[32]  J. C. Middlebrooks.  Sound localization. , 2015, Handbook of clinical neurology.

[33]  Scott P. Johnson,et al.  Preverbal Infants’ Sensitivity to Synaesthetic Cross-Modality Correspondences , 2010, Psychological science.

[34]  Ellen Winner,et al.  "Metaphorical" Mapping in Human Infants , 1981 .

[35]  Lawrence E. Marks,et al.  Synesthesia: Strong and Weak , 2001 .

[36]  L. Marks.  On colored-hearing synesthesia: cross-modal translations of sensory dimensions. , 1975, Psychological bulletin.

[37]  Daphne Maurer,et al.  Do small white balls squeak? Pitch-object correspondences in young children , 2004, Cognitive, affective & behavioral neuroscience.

[38]  H. Ruff,et al.  Infants' visual following: Effects of size and sound , 1984 .

[39]  Lorraine E. Bahrick,et al.  Infants' perception of substance and temporal synchrony in multimodal events* , 1983 .

[40]  Jamie Ward,et al.  Sound-Colour Synaesthesia: to What Extent Does it Use Cross-Modal Mechanisms Common to us All? , 2006, Cortex.