The "puzzle" of sensory perception: putting together multisensory information

For perceiving the environment our brain uses multiple sourcesof sensory information derived from several different modalities,including vision, touch and audition. The question how informationderived from these different sensory modalities converges in thebrain to form a coherent and robust percept is central tounderstanding the process of perception. My main research interestis the study of human perception focusing on multimodal integrationand visual-haptic interaction. For this, I use quantitativecomputational/statistical models together with psychophysical andneuropsychological methods. A desirable goal for the perceptual system is to maximize thereliability of the various perceptual estimates. From a statisticalviewpoint the optimal strategy for achieving this goal is tointegrate all available sensory information. This may be done usinga "maximum-likelihood-estimation" (MLE) strategy. Then the combinedpercept will be a weighted average across the individual estimateswith weights that are proportional to their reliabilities. In a recent study we could show that humans actually integratevisual and haptic information in such a statistically optimalfashion (Ernst & Banks, Nature, 2002). Others have nowdemonstrated that this finding is true not only for the integrationacross vision and touch, but also for the integration ofinformation across and within other modalities, such as audition orvision. This suggests that maximum-likelihood-estimation is aneffective and widely used strategy exploited by the perceptualsystem. By integrating sensory information the brain may or may notloose access to the individual input signals feeding into theintegrated percept. The degree to which the original information isstill accessible defines the strength of coupling between thesignals. We found that the strengths of coupling is varyingdepending on the set of signals used; e.g. strong coupling forstereo and texture signals to slant and weak coupling for visualand haptic signals to size (Hillis, Ernst, Banks, & Landy,Science, 2002). As suggested by one of our recent learning studies,the strength of coupling, which can be modeled using Bayesianstatistics, seems to depend on the natural statisticalco-occurrence between signals (Jäkel & Ernst, inprep.) Important precondition for integrating signals is to know whichsignals derived from the different modalities belong together andhow reliable these are. Recently we could show that touch can teachthe visual modality how to interpret its signals and theirreliabilities. More specifically, we could show that by exploitingtouch we can alter visual perception of slant (Ernst, Banks &Bulthoff, Nature Neuroscience, 2000). This finding contributes to avery old debate postulating that we only perceive the world becauseof our interactions with the environment. Similarly, in one of ourlatest studies we could show that experience can change theso-called "light-from-above" prior. Prior knowledge is essentialfor the interpretation of sensory signals during perception.Consequently, with the prior change we introduced a change in theperception of shape (Adams, Graf & Ernst, Nature Neuroscience,2004). Integration is only sensible if the information sources carryredundant information. If the information sources arecomplementary, different combination strategies have to beexploited. Complementation of cross-modal information wasdemonstrated in a recent study investigating visual-haptic shapeperception (Newell, Ernst, Tjan, & Bulthoff, PsychologicalScience, 2001).