Statistical Inference as a Model for Learning in ANNs

The applications of statistical inference for learning in Artiicial Neural Networks(ANNs) are reviewed in this article. ANNs are widely used to model both low level neural activities and high level cognitive functions. Statistical inference provides an objective way to derive learning algorithms for training ANNs and to evaluate the performance of the trained ANNs. The over-tting problem can be solved by model selection methods based on either a conventional statistical approach or a Bayesian approach. The learning of ANNs can be supervised and unsupervised. To train a multi-layer ANN by supervised learning is equivalent to nonlinear regression. The ensemble methods such as bagging and arching can be applied to combine ANNs to form a new classiier or predictor with a better performance. Unsupervised learning algorithms are derived by either the Hebbian law for bottom-up self-organization or global objective functions such as entropy and mutual information for top-down self-organization. 1 The Brain and ANNs Although the brain is an extremely large and complex system, from the brain organization point of view the hierarchy of the brain can be divided into eight levels: behavior, cortex, neural circuit, neuron, microcircuit, synpase, membrane and molecule. With advanced invasive and non-invasive measurement technologies, the brain can be observed at all these levels and a huge amount of data has been accumulated. Neural computational theories have been developed to account for the complex brain functions based on the accumulated data. Neural computational theories consist of neural models, neural dynamics, and learning theories. Mathematical modeling has been applied to each level in the hierarchy of the brain. From the brain function point of view, the brain is formed by three functional levels: 1. a cognitive function level related to behavior and cortex, 2. a neural activity level related to neural circuit, neuron, microcircuit and synpase, and 3. a sub-neural level related to the membrane and molecule. In this article, we only consider the rst and second functional levels. To focus on the information processing principles of the brain, we must simplify the neurons and synapses in real neural systems. ANNs are simpliied mathematical models for neural systems formed by massively interconnected computational units running in parallel. We shall discuss the applications of ANNs at both the neural activity level and the cognitive function level. 1 2 Applications of ANNs ANNs can model the brain functions at either the neural activity level or the cognitive function level. The …