Multi-sensory learning and learning to read.

The basis of literacy acquisition in alphabetic orthographies is the learning of the associations between the letters and the corresponding speech sounds. In spite of this primacy in learning to read, there is only scarce knowledge on how this audiovisual integration process works and which mechanisms are involved. Recent electrophysiological studies of letter-speech sound processing have revealed that normally developing readers take years to automate these associations and dyslexic readers hardly exhibit automation of these associations. It is argued that the reason for this effortful learning may reside in the nature of the audiovisual process that is recruited for the integration of in principle arbitrarily linked elements. It is shown that letter-speech sound integration does not resemble the processes involved in the integration of natural audiovisual objects such as audiovisual speech. The automatic symmetrical recruitment of the assumedly uni-sensory visual and auditory cortices in audiovisual speech integration does not occur for letter and speech sound integration. It is also argued that letter-speech sound integration only partly resembles the integration of arbitrarily linked unfamiliar audiovisual objects. Letter-sound integration and artificial audiovisual objects share the necessity of a narrow time window for integration to occur. However, they differ from these artificial objects, because they constitute an integration of partly familiar elements which acquire meaning through the learning of an orthography. Although letter-speech sound pairs share similarities with audiovisual speech processing as well as with unfamiliar, arbitrary objects, it seems that letter-speech sound pairs develop into unique audiovisual objects that furthermore have to be processed in a unique way in order to enable fluent reading and thus very likely recruit other neurobiological learning mechanisms than the ones involved in learning natural or arbitrary unfamiliar audiovisual associations.

[1]  Rainer Goebel,et al.  Task‐irrelevant visual letters interact with the processing of speech sounds in heteromodal and unimodal cortex , 2008, The European journal of neuroscience.

[2]  T. Picton,et al.  Mismatch Negativity: Different Water in the Same River , 2000, Audiology and Neurotology.

[3]  P. Gribble,et al.  Temporal constraints on the McGurk effect , 1996, Perception & psychophysics.

[4]  R. Goebel,et al.  Deviant processing of letters and speech sounds as proximate cause of reading failure: a functional magnetic resonance imaging study of dyslexic children. , 2010, Brain : a journal of neurology.

[5]  U. Frauenfelder,et al.  Grapheme Context Effects on Phonemic Processing , 1989 .

[6]  D W Massaro,et al.  Perception of asynchronous and conflicting visual and auditory speech. , 1996, The Journal of the Acoustical Society of America.

[7]  T. Picton,et al.  Cortical oscillations related to processing congruent and incongruent grapheme–phoneme pairs , 2006, Neuroscience Letters.

[8]  Jörg Lewald,et al.  Spatio-temporal constraints for auditory–visual integration , 2001, Behavioural Brain Research.

[9]  D. Burnham,et al.  Impact of language on development of auditory-visual speech perception. , 2008, Developmental science.

[10]  W. Schneider,et al.  Neuroimaging studies of practice-related change: fMRI and meta-analytic evidence of a domain-general control network for learning. , 2005, Brain research. Cognitive brain research.

[11]  B. Argall,et al.  Unraveling multisensory integration: patchy organization within human STS multisensory cortex , 2004, Nature Neuroscience.

[12]  E. Schröger.  Measurement and interpretation of the mismatch negativity , 1998 .

[13]  Holger Mitterer,et al.  Coping with phonological assimilation in speech perception: Evidence for early compensation , 2003, Perception & psychophysics.

[14]  Eric Vatikiotis-Bateson,et al.  The moving face during speech communication , 1998 .

[15]  L. Blomert,et al.  Long-term cognitive dynamics of fluent reading development. , 2010, Journal of experimental child psychology.

[16]  R. Campbell,et al.  Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex , 2000, Current Biology.

[17]  R. Näätänen,et al.  The mismatch negativity in evaluating central auditory dysfunction in dyslexia , 2001, Neuroscience & Biobehavioral Reviews.

[18]  H. Wimmer,et al.  The relationship of phonemic awareness to reading acquisition: More consequence than precondition but still important , 1991, Cognition.

[19]  Holger Mitterer,et al.  Auditory cortical tuning to statistical regularities in phonology , 2005, Clinical Neurophysiology.

[20]  Bruce D. McCandliss,et al.  The visual word form area: expertise for reading in the fusiform gyrus , 2003, Trends in Cognitive Sciences.

[21]  P. Bertelson,et al.  Does awareness of speech as a sequence of phones arise spontaneously? , 1979, Cognition.

[22]  D. Share.  Phonological recoding and self-teaching: sine qua non of reading acquisition , 1995, Cognition.

[23]  Ruth Fielding-Barnsley,et al.  Phonemic awareness and letter knowledge in the child's acquisition of the alphabetic principle. , 1989 .

[24]  Riitta Hari,et al.  Audiovisual Integration of Letters in the Human Brain , 2000, Neuron.

[25]  J. Kaiser,et al.  Object Familiarity and Semantic Congruency Modulate Responses in Cortical Audiovisual Integration Areas , 2007, The Journal of Neuroscience.

[26]  Penny A. MacDonald,et al.  The role of parietal cortex in awareness of self-generated movements: a transcranial magnetic stimulation study. , 2003, Cerebral cortex.

[27]  R. Näätänen.  The Mismatch Negativity: A Powerful Tool for Cognitive Neuroscience , 1995, Ear and hearing.

[28]  K. Rayner,et al.  The psychology of reading , 1989 .

[29]  A. Amedi,et al.  Functional imaging of human crossmodal identification and object recognition , 2005, Experimental Brain Research.

[30]  T. Raij.  Patterns of Brain Activity during Visual Imagery of Letters , 1999, Journal of Cognitive Neuroscience.

[31]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[32]  Mikko Sams,et al.  Processing of changes in visual speech in the human auditory cortex. , 2002, Brain research. Cognitive brain research.

[33]  Linnea C. Ehri,et al.  Phases of development in learning to read words by sight , 1995 .

[34]  Nicholas Burnett.  Education for all : literacy for life , 2005 .

[35]  Kuniyoshi L Sakai,et al.  Learning letters in adulthood: direct visualization of cortical plasticity for forming a new link between orthography and phonology. , 2004, Neuron.

[36]  J. Lewald,et al.  Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. , 2003, Brain research. Cognitive brain research.

[37]  V. Csépe.  Dyslexia: "Different Brain, Different Behavior" , 2012 .

[38]  M Ingvar,et al.  The illiterate brain. Learning to read and write during childhood influences the functional organization of the adult brain. , 1998, Brain : a journal of neurology.

[39]  D. Bishop,et al.  Using mismatch negativity to study central auditory processing in developmental language and literacy impairments: where are we, and where should we be going? , 2007, Psychological bulletin.

[40]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[41]  E Macaluso,et al.  Spatial and temporal factors during processing of audiovisual speech: a PET study , 2004, NeuroImage.

[42]  R. Goebel,et al.  Integration of Letters and Speech Sounds in the Human Brain , 2004, Neuron.

[43]  M Coltheart,et al.  DRC: a dual route cascaded model of visual word recognition and reading aloud. , 2001, Psychological review.

[44]  M. Bonte,et al.  Deviant neurophysiological responses to phonological regularities in speech in dyslexic children , 2007, Neuropsychologia.

[45]  G. E. MacKinnon,et al.  Reading Research Advances in Theory and Practice , 1985 .

[46]  G. Calvert.  Crossmodal processing in the human brain: insights from functional neuroimaging studies. , 2001, Cerebral cortex.

[47]  E. Amenedo,et al.  MMN in the visual modality: a review , 2003, Biological Psychology.

[48]  J. Pernier,et al.  Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans. , 2002, Cerebral cortex.

[49]  C. Spence,et al.  Audio-visual simultaneity judgments , 2005, Perception & psychophysics.

[50]  R. Hari,et al.  Seeing speech: visual information from lip movements modifies activity in the human auditory cortex , 1991, Neuroscience Letters.

[51]  S Lehéricy,et al.  The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. , 2000, Brain : a journal of neurology.

[52]  Philip Lieberman,et al.  Toward an Evolutionary Biology of Language , 2006 .

[53]  David J. Lewkowicz,et al.  Perception of auditory–visual temporal synchrony in human infants. , 1996 .

[54]  S A Hillyard,et al.  An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. , 2002, Brain research. Cognitive brain research.

[55]  P. Deltenre,et al.  Mismatch negativity evoked by the McGurk–MacDonald effect: a phonetic representation within short-term memory , 2002, Clinical Neurophysiology.

[56]  P. Jusczyk.  The discovery of spoken language , 1997 .

[57]  R. Näätänen,et al.  Mismatch negativity (MMN): perspectives for application. , 2000, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[58]  Marty G. Woldorff,et al.  Selective Attention and Multisensory Integration: Multiple Phases of Effects on the Evoked Brain Activity , 2005, Journal of Cognitive Neuroscience.

[59]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[60]  M. Woldorff,et al.  Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? , 2006, Cerebral cortex.

[61]  E. Bullmore,et al.  Response amplification in sensory-specific cortices during crossmodal binding. , 1999, Neuroreport.

[62]  Denis Burnham,et al.  Auditory-visual speech integration by prelinguistic infants: perception of an emergent consonant in the McGurk effect. , 2004, Developmental psychobiology.

[63]  Rainer Goebel,et al.  Top–down task effects overrule automatic multisensory responses to letter–sound pairs in auditory association cortex , 2006, NeuroImage.

[64]  A. Ghazanfar,et al.  Is neocortex essentially multisensory? , 2006, Trends in Cognitive Sciences.

[65]  M. Giard,et al.  Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study , 1999, Journal of Cognitive Neuroscience.

[66]  Lars Muckli,et al.  Cortical Plasticity of Audio–Visual Object Representations , 2008, Cerebral cortex.

[67]  A. Fort,et al.  Is the auditory sensory memory sensitive to visual information? , 2005, Experimental Brain Research.

[68]  L. Blomert,et al.  Is there a causal link from a phonological awareness deficit to reading failure in children at familial risk for dyslexia? , 2010, Dyslexia.

[69]  Dries J. W. Froyen,et al.  Exploring the Role of Low Level Visual Processing in Letter–Speech Sound Integration: A Visual MMN Study , 2010, Frontiers in Integrative Neuroscience.

[70]  R. Ilmoniemi,et al.  Brain responses reveal the learning of foreign language phonemes. , 1999, Psychophysiology.

[71]  C. Spence,et al.  The Handbook of Multisensory Processing , 2004 .

[72]  D. Poeppel,et al.  Temporal window of integration in auditory-visual speech perception , 2007, Neuropsychologia.

[73]  J. Pernier,et al.  Early auditory-visual interactions in human cortex during nonredundant target identification. , 2002, Brain research. Cognitive brain research.

[74]  I. Liberman,et al.  1. Segmentation of the spoken word and reading acquisition , 1973 .

[75]  Rainer Goebel,et al.  The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. , 2006, Cerebral cortex.

[76]  R. Goebel,et al.  Reduced Neural Integration of Letters and Speech Sounds Links Phonological and Reading Deficits in Adult Dyslexia , 2009, Current Biology.

[77]  C. Price,et al.  The role of the posterior superior temporal sulcus in audiovisual processing. , 2008, Cerebral cortex.

[78]  Dries J. W. Froyen,et al.  The Long Road to Automation: Neurocognitive Development of Letter–Speech Sound Processing , 2009, Journal of Cognitive Neuroscience.

[79]  R. Näätänen.  The perception of speech sounds by the human brain as reflected by the mismatch negativity (MMN) and its magnetic equivalent (MMNm). , 2001, Psychophysiology.

[80]  J. Gabrieli.  Dyslexia: A New Synergy Between Education and Cognitive Neuroscience , 2009, Science.

[81]  Jean Vroomen,et al.  Do you see what you are hearing? Cross-modal effects of speech sounds on lipreading , 2010, Neuroscience Letters.

[82]  J. Ziegler,et al.  Reading acquisition, developmental dyslexia, and skilled reading across languages: a psycholinguistic grain size theory. , 2005, Psychological bulletin.

[83]  Charles A. Perfetti,et al.  Phonemic knowledge and learning to read are reciprocal: A longitudinal study of first grade children. , 1987 .

[84]  A. Scheibel,et al.  Terminal patterns in cat spinal cord. 3. Primary afferent collaterals. , 1969, Brain research.

[85]  E. Bullmore,et al.  Activation of auditory cortex during silent lipreading. , 1997, Science.

[86]  Dries J. W. Froyen,et al.  Cross-modal enhancement of the MMN to speech-sounds indicates early and automatic integration of letters and speech-sounds , 2008, Neuroscience Letters.

[87]  I. Czigler.  Visual mismatch negativity: Violation of nonattended environmental regularities , 2007 .