Learning Structures for Deep Neural Networks

In this paper, we study the automatic structure learning for deep neural networks (DNN), motivated by the observations that the performance of a deep neural network is highly sensitive to its structure and previous successes of DNN heavily depend on human experts to design the network structures. We focus on the unsupervised setting for structure learning and propose to adopt the efficient coding principle, rooted in information theory and developed in computational neuroscience, to guide the procedure of structure learning without label information. This principle suggests that a good network structure should maximize the mutual information between inputs and outputs, or equivalently maximize the entropy of outputs under mild assumptions. We further establish connections between this principle and the theory of Bayesian optimal classification, and empirically verify that larger entropy of the outputs of a deep neural network indeed corresponds to a better classification accuracy. Then as an implementation of the principle, we show that sparse coding can effectively maximize the entropy of the output signals, and accordingly design an algorithm based on global group sparse coding to automatically learn the inter-layer connection and determine the depth of a neural network. Our experiments on a public image classification dataset demonstrate that using the structure learned from scratch by our proposed algorithm, one can achieve a classification accuracy comparable to the best expertdesigned structure (i.e., convolutional neural networks (CNN)). In addition, our proposed algorithm successfully discovers the local connecWork carried out at MSRA, once submitted to International Conference on Machine Learning in 2014, Beijing, China. Copyright 2014 by the author(s). tivity (corresponding to local receptive fields in CNN) and invariance structure (corresponding to pulling in CNN), as well as achieves a good tradeoff between marginal performance gain and network depth. All of this indicates the power of the efficient coding principle, and the effectiveness of automatic structure learning.