Multisensory Representation of Frequency across Audition and Touch: High Density Electrical Mapping Reveals Early Sensory-Perceptual Coupling

The frequency of environmental vibrations is sampled by two of the major sensory systems, audition and touch, notwithstanding that these signals are transduced through very different physical media and entirely separate sensory epithelia. Psychophysical studies have shown that manipulating frequency in audition or touch can have a significant cross-sensory impact on perceived frequency in the other sensory system, pointing to intimate links between these senses during computation of frequency. In this regard, the frequency of a vibratory event can be thought of as a multisensory perceptual construct. In turn, electrophysiological studies point to temporally early multisensory interactions that occur in hierarchically early sensory regions where convergent inputs from the auditory and somatosensory systems are to be found. A key question pertains to the level of processing at which the multisensory integration of featural information, such as frequency, occurs. Do the sensory systems calculate frequency independently before this information is combined, or is this feature calculated in an integrated fashion during preattentive sensory processing? The well characterized mismatch negativity, an electrophysiological response that indexes preattentive detection of a change within the context of a regular pattern of stimulation, served as our dependent measure. High-density electrophysiological recordings were made in humans while they were presented with separate blocks of somatosensory, auditory, and audio-somatosensory “standards” and “deviants,” where the deviant differed in frequency. Multisensory effects were identified beginning at ∼200 ms, with the multisensory mismatch negativity (MMN) significantly different from the sum of the unisensory MMNs. This provides compelling evidence for preattentive coupling between the somatosensory and auditory channels in the cortical representation of frequency.

[1]  Louis D Braida,et al.  Perceptual interactions in the loudness of combined auditory and vibrotactile stimuli. , 2010, The Journal of the Acoustical Society of America.

[2]  M. Alex Meredith,et al.  Crossmodal projections from somatosensory area SIV to the auditory field of the anterior ectosylvian sulcus (FAES) in Cat: further evidence for subthreshold forms of multisensory processing , 2006, Experimental Brain Research.

[3]  Brian L Allman,et al.  Subthreshold multisensory processing in cat auditory cortex , 2009, Neuroreport.

[4]  Leslie P. Keniston,et al.  Neuroanatomical identification of crossmodal auditory inputs to interneurons in somatosensory cortex , 2010, Experimental Brain Research.

[5]  C. Spence,et al.  Audiotactile interactions in roughness perception , 2002, Experimental Brain Research.

[6]  Risto N t nen.  Attention and brain function , 1992 .

[7]  John J. Foxe,et al.  Auditory facilitation of visual-target detection persists regardless of retinal eccentricity and despite wide audiovisual misalignments , 2011, Experimental Brain Research.

[8]  Louis D Braida,et al.  Integration of auditory and vibrotactile stimuli: effects of frequency. , 2010, The Journal of the Acoustical Society of America.

[9]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[10]  E. Schröger,et al.  Is there pre-attentive memory-based comparison of pitch? , 2001, Psychophysiology.

[11]  J. Gentle,et al.  Randomization and Monte Carlo Methods in Biology. , 1990 .

[12]  K. Reinikainen,et al.  Rate effect and mismatch responses in the somatosensory system: ERP-recordings in humans , 1997, Biological Psychology.

[13]  Michael J Brammer,et al.  Crossmodal identification , 1998, Trends in Cognitive Sciences.

[14]  Dries J. W. Froyen,et al.  Cross-modal enhancement of the MMN to speech-sounds indicates early and automatic integration of letters and speech-sounds , 2008, Neuroscience Letters.

[15]  C. Escera,et al.  The accuracy of sound duration representation in the human brain determines the accuracy of behavioural perception , 2000, The European journal of neuroscience.

[16]  John J. Foxe,et al.  The development of multisensory integration in high-functioning autism: high-density electrical mapping and psychophysical measures reveal impairments in the processing of audiovisual inputs. , 2013, Cerebral cortex.

[17]  F. Perrin,et al.  Spherical splines for scalp potential and current density mapping. , 1989, Electroencephalography and clinical neurophysiology.

[18]  Marc F Joanisse,et al.  Congruency of auditory sounds and visual letters modulates mismatch negativity and P300 event-related potentials. , 2011, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[19]  John J. Foxe,et al.  Right Hemispheric Contributions to Fine Auditory Temporal Discriminations: High-Density Electrical Mapping of the Duration Mismatch Negativity (MMN) , 2009, Frontiers in integrative neuroscience.

[20]  D. Guthrie,et al.  Significance testing of difference potentials. , 1991, Psychophysiology.

[21]  Sibylle C. Herholz,et al.  Auditory‐Somatosensory Integration and Cortical Plasticity in Musical Training , 2009, Annals of the New York Academy of Sciences.

[22]  John J. Foxe,et al.  Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. , 2005, Cerebral cortex.

[23]  John J. Foxe,et al.  Common or Redundant Neural Circuits for Duration Processing across Audition and Touch , 2011, The Journal of Neuroscience.

[24]  F. Perrin,et al.  Separate Representation of Stimulus Frequency, Intensity, and Duration in Auditory Sensory Memory: An Event-Related Potential and Dipole-Model Analysis , 1995, Journal of Cognitive Neuroscience.

[25]  Ankoor S. Shah,et al.  Auditory Cortical Neurons Respond to Somatosensory Stimulation , 2003, The Journal of Neuroscience.

[26]  Leslie P. Keniston,et al.  Auditory influences on non-auditory cortices , 2009, Hearing Research.

[27]  John J. Foxe,et al.  Multisensory Integration: Frequency Tuning of Audio-Tactile Integration , 2009, Current Biology.

[28]  Sidney S. Simon,et al.  Merging of the Senses , 2008, Front. Neurosci..

[29]  I. Winkler,et al.  MMN or no MMN: no magnitude of deviance effect on the MMN amplitude. , 2007, Psychophysiology.

[30]  M. Wallace.  The Development of Multisensory Integration. , 2004 .

[31]  J. Driver,et al.  Multisensory Interplay Reveals Crossmodal Influences on ‘Sensory-Specific’ Brain Regions, Neural Responses, and Judgments , 2008, Neuron.

[32]  Risto Näätänen,et al.  The additivity of the auditory feature analysis in the human brain as indexed by the mismatch negativity: 1+1≈2 but 1+1+1<3 , 2001, Neuroscience Letters.

[33]  S. S. Stevens.  The Volume and Intensity of Tones , 1934 .

[34]  John J. Foxe,et al.  Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. , 2000, Brain research. Cognitive brain research.

[35]  G. V. Simpson,et al.  Flow of activation from V1 to frontal cortex in humans , 2001, Experimental Brain Research.

[36]  John J. Foxe,et al.  The development of audiovisual multisensory integration across childhood and early adolescence: a high-density electrical mapping study. , 2011, Cerebral cortex.

[37]  John J. Foxe,et al.  The case for feedforward multisensory convergence during early cortical processing , 2005, Neuroreport.

[38]  D. Lehmann,et al.  Reference-free identification of components of checkerboard-evoked multichannel potential fields. , 1980, Electroencephalography and clinical neurophysiology.

[39]  V. Jousmäki,et al.  Parchment-skin illusion: sound-biased touch , 1998, Current Biology.

[40]  John J. Foxe,et al.  The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. , 2002, Brain research. Cognitive brain research.

[41]  I. Winkler,et al.  Mismatch negativity to pitch change: varied stimulus proportions in controlling effects of neural refractoriness on human auditory event-related brain potentials , 2003, Neuroscience Letters.

[42]  John J. Foxe,et al.  Multisensory contributions to low-level, ‘unisensory’ processing , 2005, Current Opinion in Neurobiology.

[43]  I. Winkler,et al.  Deviance detection in congruent audiovisual speech: Evidence for implicit integrated audiovisual memory representations , 2009, Biological Psychology.

[44]  John J. Foxe,et al.  Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion , 2007, Neuropsychologia.

[45]  Mikko Sams,et al.  Processing of changes in visual speech in the human auditory cortex. , 2002, Brain research. Cognitive brain research.

[46]  John J. Foxe,et al.  The neural circuitry of pre-attentive auditory change-detection: an fMRI study of pitch and duration mismatch negativity generators. , 2005, Cerebral cortex.

[47]  J. F. Dammann,et al.  Temporal Frequency Channels Are Linked across Audition and Touch , 2009, Current Biology.

[48]  Erich Schröger,et al.  Human pre-attentive auditory change-detection with single, double, and triple deviations as revealed by mismatch negativity additivity , 2001, Neuroscience Letters.

[49]  R Näätänen,et al.  The additivity of the auditory feature analysis in the human brain as indexed by the mismatch negativity: 1+1 approximately 2 but 1+1+1<3. , 2001, Neuroscience letters.

[50]  Louis D Braida,et al.  Integration of auditory and vibrotactile stimuli: effects of phase and stimulus-onset asynchrony. , 2009, The Journal of the Acoustical Society of America.

[51]  John J. Foxe,et al.  Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study. , 2002, Journal of neurophysiology.

[52]  F. Perrin,et al.  Mapping of scalp potentials by surface spline interpolation. , 1987, Electroencephalography and clinical neurophysiology.

[53]  R. Näätänen,et al.  The mismatch negativity (MMN) in basic research of central auditory processing: A review , 2007, Clinical Neurophysiology.

[54]  C. Schroeder,et al.  Somatosensory input to auditory association cortex in the macaque monkey. , 2001, Journal of neurophysiology.