Online Representation Search and Its Interactions with Unsupervised Learning

We consider the problem of finding good hidden units, or features, for use in multilayer neural networks. Solution methods that generate candidate features, evaluate them, and retain the most useful ones (such as cascade correlation and NEAT), we call representation search methods. In this paper, we explore novel representation search methods in an online setting, compare them with two simple unsupervised learning algorithms that also scale online. We demonstrate that the unsupervised learning methods are effective only at the initial learning period. However, when combined with search strategies, they are able to improve representation with more data and perform better than either of search and unsupervised learning alone. We conclude that search has enabling effects on unsupervised learning in continual learning tasks.

[1]  Leonard Uhr,et al.  A pattern recognition program that generates, evaluates, and adjusts its own operators , 1961, IRE-AIEE-ACM '61 (Western).

[2]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[3]  A. Klopf,et al.  An Evolutionary Pattern Recognition Network , 1969 .

[4]  Pentti Kanerva,et al.  Sparse Distributed Memory , 1988 .

[5]  Richard W. Prager,et al.  The modified Kanerva model for automatic speech recognition , 1989 .

[6]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[7]  Richard S. Sutton,et al.  Online Learning with Random Representations , 1993, ICML.

[8]  Alan F. Murray,et al.  IEEE International Conference on Neural Networks , 1997 .

[9]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[10]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[11]  Yann LeCun,et al.  What is the best multi-stage architecture for object recognition? , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[12]  David D. Cox,et al.  A High-Throughput Screening Approach to Discovering Good Forms of Biologically Inspired Visual Representation , 2009, PLoS Comput. Biol..

[13]  Zhenghao Chen,et al.  On Random Weights and Unsupervised Feature Learning , 2011, ICML.

[14]  Patrick M. Pilarski,et al.  Tuning-free step-size adaptation , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).