Conscious access to the unisensory components of a cross-modal illusion

It is often claimed that binding information across sensory modalities leads to coherent, unitary mental representations. The dramatic illusions experienced as a result of intersensory conflict, such as the McGurk effect, are often attributed to a propensity of the perceptual system to impose multisensory coherence onto events originating from a common source. In contrast with this ssumption of unity, we report an unexpected ability to resolve the timing between sound and sight regarding multisensory events that induce an illusory reversal of the elements specified in each modality. This finding reveals that the brain can gain access, simultaneously, to unisensory component information as well as to the result of the integrated multisensory percept, suggesting some degree of penetrability in the processes leading to cross-modality binding.

[1]  Alan Kingstone,et al.  Assessing automaticity in the audiovisual integration of motion. , 2005, Acta psychologica.

[2]  Marty G. Woldorff,et al.  Selective Attention and Multisensory Integration: Multiple Phases of Effects on the Evoked Brain Activity , 2005, Journal of Cognitive Neuroscience.

[3]  R. Hari,et al.  Seeing speech: visual information from lip movements modifies activity in the human auditory cortex , 1991, Neuroscience Letters.

[4]  G. Calvert Crossmodal processing in the human brain: insights from functional neuroimaging studies. , 2001, Cerebral cortex.

[5]  P. Bertelson,et al.  The psychology of multimodal perception , 2004 .

[6]  A. Fort,et al.  Bimodal speech: early suppressive visual effects in human auditory cortex , 2004, The European journal of neuroscience.

[7]  Mikko Sams,et al.  Audio–visual speech perception is special , 2005, Cognition.

[8]  J. Navarra,et al.  Assessing automaticity in audiovisual speech integration: evidence from the speeded classification task , 2004, Cognition.

[9]  Sidney S. Simon,et al.  Merging of the Senses , 2008, Front. Neurosci..

[10]  M. Hallett,et al.  Neural Correlates of Auditory–Visual Stimulus Onset Asynchrony Detection , 2001, The Journal of Neuroscience.

[11]  M. Sams,et al.  Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. , 2003, Brain research. Cognitive brain research.

[12]  John J. Foxe,et al.  Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. , 2006, Cerebral cortex.

[13]  E. Bullmore,et al.  Activation of auditory cortex during silent lipreading. , 1997, Science.

[14]  B. Weintraub,et al.  Secretion of hCG-α subunit and hCG by HeLa strains , 1977, Nature.

[15]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[16]  Lee M. Miller,et al.  Behavioral/systems/cognitive Perceptual Fusion and Stimulus Coincidence in the Cross- Modal Integration of Speech , 2022 .

[17]  C. Spence,et al.  Crossmodal Space and Crossmodal Attention , 2004 .

[18]  J. Fodor The Modularity of mind. An essay on faculty psychology , 1986 .

[19]  Mark T. Wallace,et al.  Crossmodal spatial interactions in subcortical and cortical circuits , 2004 .

[20]  P. Bertelson,et al.  The ventriloquist effect does not depend on the direction of deliberate visual attention , 2000, Perception & psychophysics.

[21]  Joh. Müller,et al.  Handbuch der Physiologie des Menschen für Vorlesungen , 1835 .

[22]  J. Driver Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading , 1996, Nature.

[23]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[24]  R. Campbell,et al.  Audiovisual Integration of Speech Falters under High Attention Demands , 2005, Current Biology.

[25]  P. Gribble,et al.  Temporal constraints on the McGurk effect , 1996, Perception & psychophysics.

[26]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[27]  Z. Pylyshyn Is vision continuous with cognition? The case for cognitive impenetrability of visual perception. , 1999, The Behavioral and brain sciences.