Crossmodal interactions: lessons from synesthesia.

Synesthesia is a condition in which stimulation in one modality also gives rise to a perceptual experience in a second modality. In two recent studies we found that the condition is more common than previously reported; up to 5% of the population may experience at least one type of synesthesia. Although the condition has been traditionally viewed as an anomaly (e.g., breakdown in modularity), it seems that at least some of the mechanisms underlying synesthesia do reflect universal crossmodal mechanisms. We review here a number of examples of crossmodal correspondences found in both synesthetes and nonsynesthetes including pitch-lightness and vision-touch interaction, as well as cross-domain spatial-numeric interactions. Additionally, we discuss the common role of spatial attention in binding shape and color surface features (whether ordinary or synesthetic color). Consistently with behavioral and neuroimaging data showing that chromatic-graphemic (colored-letter) synesthesia is a genuine perceptual phenomenon implicating extrastriate cortex, we also present electrophysiological data showing modulation of visual evoked potentials by synesthetic color congruency.

[1]  Jamie Ward,et al.  Is Synaesthesia an X-Linked Dominant Trait with Lethality in Males? , 2005, Perception.

[2]  Lawrence E. Marks,et al.  Synesthesia: Strong and Weak , 2001 .

[3]  L. Marks On colored-hearing synesthesia: cross-modal translations of sensory dimensions. , 1975, Psychological bulletin.

[4]  Lotfi B Merabet,et al.  Visual Hallucinations During Prolonged Blindfolding in Sighted Subjects , 2004, Journal of neuro-ophthalmology : the official journal of the North American Neuro-Ophthalmology Society.

[5]  Lynn C. Robertson,et al.  Space, Objects, Minds and Brains , 2003 .

[6]  E. Bullmore,et al.  Functional magnetic resonance imaging , 2001 .

[7]  L. Hollister,et al.  Effect of mescaline, lysergic acid diethylamide and psilocybin on color perception , 1963, Psychopharmacologia.

[8]  M. Graziano Where is my arm? The relative role of vision and proprioception in the neuronal representation of limb position. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[9]  S. Dehaene,et al.  The mental representation of parity and number magnitude. , 1993 .

[10]  Daniel Smilek,et al.  Not all synaesthetes are created equal: Projector versus associator synaesthetes , 2004, Cognitive, Affective, & Behavioral Neuroscience.

[11]  John C. Marshall,et al.  When seeing is feeling: Acquired synaesthesia or phantom touch? , 1996 .

[12]  Simon Baron-Cohen,et al.  Synaesthesia: Prevalence and Familiality , 1996, Perception.

[13]  C. Maitland,et al.  Auditory-visual synesthesia. Report of a case with intact visual pathways. , 1984, Archives of neurology.

[14]  Jamie Ward,et al.  Emotionally mediated synaesthesia , 2004, Cognitive neuropsychology.

[15]  Wim Fias,et al.  The mental representation of ordinal sequences is spatially organized , 2003, Cognition.

[16]  C. Frith,et al.  Experiencing Oneself vs Another Person as Being the Cause of an Action: The Neural Correlates of the Experience of Agency , 2002, NeuroImage.

[17]  S Baron-Cohen,et al.  The physiology of coloured hearing. A PET activation study of colour-word synaesthesia. , 1995, Brain : a journal of neurology.

[18]  T. Sejnowski,et al.  Perspectives on cognitive neuroscience. , 1988, Science.

[19]  G. Fink,et al.  Cerebral Representation of One’s Own Past: Neural Networks Involved in Autobiographical Memory , 1996, The Journal of Neuroscience.

[20]  Lawrence E. Marks,et al.  On colored-hearing synesthesia: cross-modal translations of sensory dimensions. , 1975 .

[21]  Simon Baron-Cohen,et al.  Synaesthesia : classic and contemporary readings , 1997 .

[22]  Lynn C. Robertson,et al.  What Can Spatial Deficits Teach Us About Feature Binding and Spatial Maps , 1999 .

[23]  Georg Goldenberg,et al.  Imagery without perception—A case study of anosognosia for cortical blindness , 1995, Neuropsychologia.

[24]  E. Bullmore,et al.  Activation of auditory cortex during silent lipreading. , 1997, Science.

[25]  N. Sagiv,et al.  What is the relationship between synaesthesia and visuo-spatial number forms? , 2006, Cognition.

[26]  Avishai Henik,et al.  When Blue is Larger than Red: Colors Influence Numerical Cognition in Synesthesia , 2005, Journal of Cognitive Neuroscience.

[27]  R. Cytowič,et al.  Synesthesia: Phenomenology And Neuropsychology , 2008 .

[28]  B. Shanon,et al.  Colour associates to semantic linear orders , 1982, Psychological research.

[29]  D. Maurer Neonatal Synesthesia: Implications for the Processing of Speech And Faces , 1993 .

[30]  Lorena R. R. Gianotti,et al.  Synesthesia: when colors count. , 2005, Brain research. Cognitive brain research.

[31]  B. Laeng,et al.  Does Color Synesthesia Pose a Paradox for Early-Selection Theories of Attention? , 2004, Psychological science.

[32]  Geoffrey M. Boynton,et al.  Individual Differences among Grapheme-Color Synesthetes: Brain-Behavior Correlations , 2005, Neuron.

[33]  L. Jacobs,et al.  Auditory-visual synesthesia: sound-induced photisms. , 1981, Archives of neurology.

[34]  B. Shanon The Antipodes of the Mind: Charting the Phenomenology of the Ayahuasca Experience , 2003 .

[35]  H. Neville Developmental specificity in neurocognitive development in humans. , 1995 .

[36]  N F Ramsey,et al.  Activation of striate cortex in the absence of visual stimulation: an fMRI study of synesthesia , 2001, Neuroreport.

[37]  M. Mon-Williams,et al.  Synaesthesia in the normal limb , 1997, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[38]  E. Bizzi,et al.  The Cognitive Neurosciences , 1996 .

[39]  R. Walsh Can Synaesthesia Be Cultivated? Indications from Surveys of Meditators , 2005 .

[40]  V. Ramachandran,et al.  Synaesthesia in phantom limbs induced with mirrors , 1996, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[41]  Charles S. Myers,et al.  A CASE OF SYNAESTHESIA , 1911 .

[42]  L E Marks,et al.  On associations of light and sound: the mediation of brightness, pitch, and loudness. , 1974, The American journal of psychology.

[43]  C. Spence Multisensory attention and tactile information-processing , 2002, Behavioural Brain Research.

[44]  T L Hubbard,et al.  Synesthesia-like mappings of lightness, pitch, and melodic interval. , 1996, The American journal of psychology.

[45]  Jeffrey Heer,et al.  Does Binding of Synesthetic Color to the Evoking Grapheme Require Attention? , 2006, Cortex.

[46]  C. Blakemore,et al.  Visual Synaesthesia in the Blind , 2004, Perception.

[47]  A. Treisman,et al.  Parietal contributions to visual feature binding: evidence from a patient with bilateral lesions , 1995, Science.

[48]  W. Singer,et al.  Neuronal Correlates of Colour-Graphemic Synaesthesia: Afmri Study , 2006, Cortex.

[49]  Richard S. J. Frackowiak,et al.  Cross-Modal Plasticity Underpins Language Recovery after Cochlear Implantation , 2001, Neuron.

[50]  Jamie Ward,et al.  Sound-Colour Synaesthesia: to What Extent Does it Use Cross-Modal Mechanisms Common to us All? , 2006, Cortex.

[51]  S. Baron-Cohen,et al.  Coloured Speech Perception: Is Synaesthesia what Happens when Modularity Breaks Down? , 1993, Perception.

[52]  H. Kennedy,et al.  Characterization of transient cortical projections from auditory, somatosensory, and motor cortices to visual areas 17, 18, and 19 in the kitten , 1988, The Journal of comparative neurology.

[53]  Lynn C. Robertson,et al.  Synesthesia : perspectives from cognitive neuroscience , 2005 .

[54]  V. Ramachandran,et al.  Synaesthesia? A window into perception, thought and language , 2001 .

[55]  Randolph Blake,et al.  The perceptual reality of synesthetic colors , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[56]  L E Marks,et al.  On cross-modal similarity: auditory-visual interactions in speeded discrimination. , 1987, Journal of experimental psychology. Human perception and performance.

[57]  Richard E. Cytowic Synesthesia: A union of the senses, 2nd ed. , 2002 .

[58]  L. Fogassi,et al.  A Touching Sight SII/PV Activation during the Observation and Experience of Touch , 2004, Neuron.

[59]  Daniel Smilek,et al.  The Importance of Individual Differences in Grapheme-Color Synesthesia , 2005, Neuron.

[60]  Marie-Pascale Noël,et al.  Images of numbers, or “when 98 is upper left and 6 sky blue” , 1992, Cognition.

[61]  Patrik Vuilleumier,et al.  The Number Space and Neglect , 2004, Cortex.

[62]  H. Emrich,et al.  Neurophysiological aspects of synesthetic experience. , 1999, The Journal of neuropsychiatry and clinical neurosciences.

[63]  Daniel Smilek,et al.  Seeing double: The role of meaning in alphanumeric-colour synaesthesia , 2003, Brain and Cognition.

[64]  E. Macaluso,et al.  Multisensory spatial interactions: a window onto functional integration in the human brain , 2005, Trends in Neurosciences.

[65]  V. Ramachandran,et al.  Psychophysical investigations into the neural basis of synaesthesia , 2001, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[66]  Michael D. Dodd,et al.  Perceiving numbers causes spatial shifts of attention , 2003, Nature Neuroscience.

[67]  Vilayanur S. Ramachandran,et al.  Acquired synesthesia in retinitis pigmentosa , 1999 .

[68]  Anina N. Rich,et al.  A systematic, large-scale study of synaesthesia: implications for the role of early experience in lexical-colour associations , 2005, Cognition.

[69]  G. Innocenti,et al.  Development of projections from auditory to visual areas in the cat , 1988, The Journal of comparative neurology.

[70]  Daniel Smilek,et al.  Synaesthetic Photisms Influence Visual Perception , 2001, Journal of Cognitive Neuroscience.

[71]  Vincent Walsh A theory of magnitude: common cortical metrics of time, space and quantity , 2003, Trends in Cognitive Sciences.

[72]  R. Salmelin,et al.  Dynamics of letter string perception in the human occipitotemporal cortex. , 1999, Brain : a journal of neurology.

[73]  L. Robertson Binding, spatial attention and perceptual awareness , 2003, Nature Reviews Neuroscience.

[74]  J. Wolfe,et al.  The Psychophysical Evidence for a Binding Problem in Human Vision , 1999, Neuron.

[75]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[76]  A. Treisman,et al.  Illusory conjunctions in the perception of objects , 1982, Cognitive Psychology.

[77]  Henry Kennedy,et al.  Synaesthesia: Implications for developmental neurobiology. , 1997 .

[78]  Jamie Ward,et al.  Non-random associations of graphemes to colours in synaesthetic and non-synaesthetic populations , 2005, Cognitive neuropsychology.

[79]  E. T. Bullmore,et al.  Functional magnetic resonance imaging of synesthesia: activation of V4/V8 by spoken words , 2002, Nature Neuroscience.

[80]  A. King,et al.  Multisensory integration: perceptual grouping by eye and ear , 2001, Current Biology.

[81]  Jamie Ward,et al.  Synaesthesia: The Prevalence of Atypical Cross-Modal Experiences , 2006, Perception.

[82]  J. Pernier,et al.  ERP Manifestations of Processing Printed Words at Different Psycholinguistic Levels: Time Course and Scalp Distribution , 1999, Journal of Cognitive Neuroscience.

[83]  H. Kennedy,et al.  Anatomical Evidence of Multimodal Integration in Primate Striate Cortex , 2002, The Journal of Neuroscience.

[84]  C. Brinkman,et al.  Do Synaesthetic Colours Act as Unique Features in Visual Search? , 2006, Cortex.

[85]  S-J Blakemore,et al.  Somatosensory activations during the observation of touch and a case of vision-touch synaesthesia. , 2005, Brain : a journal of neurology.

[86]  FRANCIS GALTON,et al.  Visualised Numerals , 1880, Nature.

[87]  S. Dehaene,et al.  Interactions between number and space in parietal cortex , 2005, Nature Reviews Neuroscience.

[88]  Simon Baron-Cohen,et al.  Is There a Normal Phase of Synaesthesia in Development , 1996 .

[89]  C. Mills,et al.  DIGIT SYNAESTHESIA: A CASE STUDY USING A STROOP-TYPE TEST , 1999 .

[90]  P. Haggard,et al.  Noninformative vision improves the spatial resolution of touch in humans , 2001, Current Biology.