Self-organization of neural maps using a modulated BCM rule within a multimodal architecture

Human beings interact with the environment through different modalities, i.e. perceptions and actions. Different perceptions as view, audition or proprioception for example, are picked up by different spatially separated sensors. They are processed in the cortex by dedicated brain areas, which are self-organized, so that spatially close neurons are sensitive to close stimuli. However, the processings of these perceptive flows are not isolated. On the contrary, they are constantly interacting, as illustrated by the McGurk effect. When the phonetic stimulus /ba/ is presented simultaneously with a lip movement corresponding to a /ga/, people perceive a /da/, which does not correspond to any of the stimuli. Merging several stimuli into one multimodal perception reduces the ambiguities and the noise of each perception. This is an essential mechanism of the cortex to interact with the environment. The aim of this article is to propose a model for the assembling of modalities, inspired by the biological properties of the cortex. We have modified the Bienenstock Cooper Munro (BCM) rule to include it in a model that consists of interacting maps of multilayer cortical columns. Each map is able to self-organize thanks to a continuous decentralized and local learning modulated by a high level signal. By assembling different maps corresponding to different modalities, our model creates a multimodal context which is used as a modulating signal and thus it influences the self-organization of each map.