A multimodal self-organizing network for sensory integration of letters and phonemes

Integration of signals from sensory receptors of different modalities is known to enhance perception. Integration takes place in bimodal and multimodal association areas of neocortex and results in robust bimodal or multimodal percepts as well as in feedback mediated enhanced unimodal perception. A Multimodal Self-Organizing Network, MuSON, is presented as a tool for simulating sensory integration. The latest version of this MuSON, that is described in the current paper, also takes the degree of recognition of stimuli in the various maps of the network into account. Simulation results show the same characteristics as corresponding results from psychology and neuroscience.

[1]  Willy Serniclaes,et al.  Neural correlates of switching from auditory to speech perception , 2005, NeuroImage.

[2]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[3]  Harry F. Olson,et al.  Phonetic typewriter , 1957 .

[4]  Teuvo Kohonen,et al.  The 'neural' phonetic typewriter , 1988, Computer.

[5]  V. Lamme,et al.  The distinct modes of vision offered by feedforward and recurrent processing , 2000, Trends in Neurosciences.

[6]  Jon Driver,et al.  Crossmodal attention , 1998, Current Opinion in Neurobiology.

[7]  T. Hackett,et al.  Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[8]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[9]  B. Repp,et al.  Can speech perception be influenced by simultaneous presentation of print , 1988 .

[10]  Andrew P. Paplinski,et al.  Speaker-dependent Bimodal Integration of Chinese Phonemes and Letters Using Multimodal Self-organizing Networks , 2007, 2007 International Joint Conference on Neural Networks.

[11]  Andrew P. Paplinski,et al.  Bimodal Integration of Phonemes and Letters: an Application of Multimodal Self-Organizing Networks , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[12]  E. T. Possing,et al.  Human temporal lobe activation by speech and nonspeech sounds. , 2000, Cerebral cortex.

[13]  E. Bullmore,et al.  Activation of auditory cortex during silent lipreading. , 1997, Science.

[14]  John J. Foxe,et al.  The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. , 2002, Brain research. Cognitive brain research.

[15]  T A Polk,et al.  The neural development and organization of letter recognition: evidence from functional neuroimaging, computational modeling, and behavioral studies. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[16]  C. Price.  The anatomy of language: contributions from functional neuroimaging , 2000, Journal of anatomy.

[17]  Andrew P. Paplinski,et al.  Multimodal FeedForward Self-organizing Maps , 2005, CIS.

[18]  R. Campbell,et al.  Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex , 2000, Current Biology.

[19]  M. Farah,et al.  Neural Specialization for Letter Recognition , 2002, Journal of Cognitive Neuroscience.

[20]  Mikko Sams,et al.  Perceiving identical sounds as speech or non-speech modulates activity in the left posterior superior temporal sulcus , 2006, NeuroImage.

[21]  Riitta Hari,et al.  Audiovisual Integration of Letters in the Human Brain , 2000, Neuron.

[22]  R. Campbell,et al.  Reading Speech from Still and Moving Faces: The Neural Substrates of Visible Speech , 2003, Journal of Cognitive Neuroscience.

[23]  G. Calvert,et al.  Multisensory integration: methodological approaches and emerging principles in the human brain , 2004, Journal of Physiology-Paris.

[24]  M. Tarr,et al.  The Fusiform Face Area is Part of a Network that Processes Faces at the Individual Level , 2000, Journal of Cognitive Neuroscience.

[25]  Ulrich H. Frauenfelder,et al.  Bidirectional grapheme-phoneme activation in a bimodal detection task. , 1993, Journal of experimental psychology. Human perception and performance.

[26]  R. Goebel,et al.  Integration of Letters and Speech Sounds in the Human Brain , 2004, Neuron.

[27]  John J. Foxe,et al.  The case for feedforward multisensory convergence during early cortical processing , 2005, Neuroreport.

[28]  C. Spence,et al.  The Handbook of Multisensory Processing , 2004 .

[29]  Andrew P. Paplinski,et al.  Feedback in Multimodal Self-organizing Networks Enhances Perception of Corrupted Stimuli , 2006, Australian Conference on Artificial Intelligence.