Parametric Statistical Estimation with Artificial Neural Networks: A Condensed Discussion

Learning in artificial neural networks is a process by which experience arising from exposure to measurements of empirical phenomena is converted to knowledge, embodied in network weights. This process can be viewed formally as statistical estimation of the parameters of a parametrized probability model. We exploit this formal viewpoint to give a unified theory of learng in artificial neural networks. The theory encompasses both supervised and unsupervised learning in either feedforward or recurrent networks. We begin by describing various objects appropriate for learning, such as conditional means, variances or quantiles, or conditional densities. We then show how artificial neural networks can be viewed as parametric statistical models directed toward these objects of interest. We show how a probability density can be associated with the output of any network, and use this density to define network weights indexing an information-theoretically optimal approximation to the object of interest. We next study statistical properties of quasi-maximum likelihood estimators consistent for the optimal weights, including issues associated with statistical inference about the optimal weights. Finally, we consider computational methods for obtaining these estimators, with special attention to extensions of the method of back-propagation.