Semilinear Predictability Minimization Produces Well-Known Feature Detectors

Predictability minimization (PMSchmidhuber 1992) exhibits various intuitive and theoretical advantages over many other methods for unsupervised redundancy reduction. So far, however, there have not been any serious practical applications of PM. In this paper, we apply semilinear PM to static real world images and find that without a teacher and without any significant preprocessing, the system automatically learns to generate distributed representations based on well-known feature detectors, such as orientation-sensitive edge detectors and off-centeron-surround detectors, thus extracting simple features related to those considered useful for image preprocessing and compression.

[1]  R Linsker,et al.  From basic network principles to neural architecture: emergence of orientation columns. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[2]  R. Linsker,et al.  From basic network principles to neural architecture , 1986 .

[3]  R Linsker,et al.  From basic network principles to neural architecture: emergence of spatial-opponent cells. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[4]  Ralph Linsker,et al.  Self-organization in a perceptual network , 1988, Computer.

[5]  J. Rubner,et al.  A Self-Organizing Network for Principal-Component Analysis , 1989 .

[6]  H. B. Barlow,et al.  Finding Minimum Entropy Codes , 1989, Neural Computation.

[7]  J. Rubner,et al.  Development of feature detectors by self-organization. A network model. , 1990, Biological cybernetics.

[8]  D. Mackay,et al.  Analysis of Linsker's application of Hebbian rules to linear networks , 1990 .

[9]  Zhaoping Li,et al.  Understanding Retinal Color Coding from First Principles , 1992, Neural Computation.

[10]  Jürgen Schmidhuber,et al.  Learning Factorial Codes by Predictability Minimization , 1992, Neural Computation.

[11]  Jürgen Schmidhuber,et al.  Netzwerkarchitekturen, Zielfunktionen und Kettenregel , 1993 .

[12]  Stefanie N. Lindstaedt,et al.  Comparison of two Unsupervised Neural Network Models for Redundancy Reduction , 1993 .

[13]  KD Miller.  A model for the development of simple cell receptive fields and the ordered arrangement of orientation columns through activity-dependent competition between ON- and OFF-center inputs , 1994, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[14]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[15]  G. Deco,et al.  An Information-Theoretic Approach to Neural Computing , 1997, Perspectives in Neural Computing.

[16]  G. Sperling,et al.  Structure detection: a statistically certified unsupervised learning procedure , 1997, Vision Research.

[17]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[18]  G. Krieger,et al.  Neural network models and the visual cortex: the missing link between orientation selectivity and the natural environment , 1997, Neuroscience Letters.

[19]  Corso Elvezia.  Neural Predictors for Detecting and Removing Redundant Information , 1998 .