Maximizing Mutual Information

Introduction Consider the problem of getting a neural network to associate an appropriate response with an image sequence. The obvious approach is to use supervised training. If the network has around 1014 parameters and bnly lives for around lo9 seconds, the supervision signal had better contain at least lo5 bits per second to make use of the capacity of the synapses. It is not immediately obvious where such a rich supervision signal could come from. A more promising approach depends on the observation that images are not random but are generated by physical processes of limited complexity and that the appropriate response to an image nearly always depends on the physical causes of the image rather than the pixel intensities. This suggests that an unsupervised learning process should be used to solve the difficult problem of extracting the underlying causes, and decisions about responses can be left to a separate learning algorithm that takes the underlying causes rather than the saw sensory data as its inputs. Unsupervised learning can usually be viewed as a method of modeling the probability density of the inputs, so the rich sensory input itself can provide the lo5 bits per second of constraint that is required to amke use of the capacity of the synapses. The papers in this collection provide a sample of research on unsuper-vised learning. Some areas and important contributions are not represented either because an appropriate paper did not appear in Neural Computation or because of the limited space that was available. One entire area of research in unsupervised learning, self-organizing map formation, will appear as a separate volume in this series. Despite these limitations, the wide range of approaches that is included here serves as a guide to the development of the field of unsupervised learning. Redundancy Reduction One of the earliest formulations of unsupervised learning in the context of vision was the concept of redundancy reduction (Attneave 1954; Barlow 1959; Barlow 1989). The goal was to find ways to compress the information contained in images, a goal that was also pursued in the commercial arena to reduce the bandwidth needed to transmit images. In the case of the human visual system, information in the array of photoreceptors in the retina, which number around 100 million, is compressed and represented by spike trains in around 1 million ganglion cells whose axons form the optic nerve. Atick and Redlich (1993) used …