Learning the parts of objects by non-negative matrix factorization

Is perception of the whole based on perception of its parts? There is psychological and physiological evidence for parts-based representations in the brain, and certain computational theories of object recognition rely on such representations. But little is known about how brains or computers might learn the parts of objects. Here we demonstrate an algorithm for non-negative matrix factorization that is able to learn parts of faces and semantic features of text. This is in contrast to other methods, such as principal components analysis and vector quantization, that learn holistic, not parts-based, representations. Non-negative matrix factorization is distinguished from the other methods by its use of non-negativity constraints. These constraints lead to a parts-based representation because they allow only additive, not subtractive, combinations. When non-negative matrix factorization is implemented as a neural network, parts-based representations emerge by virtue of two properties: the firing rates of neurons are never negative and synaptic strengths do not change sign.

[1]  G. J. Hinde,et al.  On the Sponge-remains in the Lower Tertiary Strata near Oamaru, Otago, New Zealand. , 1892 .

[2]  William H. Richardson,et al.  Bayesian-Based Iterative Method of Image Restoration , 1972 .

[3]  L. Lucy An iterative technique for the rectification of observed distributions , 1974 .

[4]  H. Harper,et al.  Silica, diatoms, and Cenozoic radiolarian evolution , 1975 .

[5]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[6]  S. Palmer Hierarchical structure in perceptual representation , 1977, Cognitive Psychology.

[7]  F. Harrison Living and Fossil Sponges: Notes for a Short Course.Willard D. Hartman , Jobst W. Wendt , Felix Wiedenmayer , Robert N. Ginsburg , Pamela Reid , 1981 .

[8]  T. J. Palmer,et al.  Ecology of sponge reefs from the Upper Bathonian of Normandy , 1981 .

[9]  L. Shepp,et al.  Maximum Likelihood Reconstruction for Emission Tomography , 1983, IEEE Transactions on Medical Imaging.

[10]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[11]  I. Biederman Recognition-by-components: a theory of human image understanding. , 1987, Psychological review.

[12]  J. Vacelet Indications de profondeur données par les Spongiaires dans les milieux benthiques actuels , 1988 .

[13]  A. Knoll,et al.  Secular change in chert distribution: a reflection of evolving biological participation in the silica cycle. , 1989, Palaios.

[14]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[15]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[16]  K Nakayama,et al.  Experiencing and perceiving visual surfaces. , 1992, Science.

[17]  R. Leinfelder Upper Jurassic reef types and controlling factors A preliminary report , 1993 .

[18]  M. Kelly-Borges,et al.  Phylogeny and classification of lithistid sponges (porifera: Demospongiae): a preliminary assessment using ribosomal DNA sequence comparisons. , 1994, Molecular marine biology and biotechnology.

[19]  D. Perrett,et al.  Recognition of objects and their component parts: responses of single units in the temporal cortex of the macaque. , 1994, Cerebral cortex.

[20]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[21]  D. M. Nelson,et al.  Production and dissolution of biogenic silica in the ocean: Revised global estimates, comparison with regional data and relationship to biogenic sedimentation , 1995 .

[22]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[23]  D. M. Nelson,et al.  The Silica Balance in the World Ocean: A Reestimate , 1995, Science.

[24]  Geoffrey E. Hinton,et al.  The "wake-sleep" algorithm for unsupervised neural networks. , 1995, Science.

[25]  H. Sebastian Seung,et al.  Unsupervised Learning by Convex and Conic Coding , 1996, NIPS.

[26]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[27]  S. Ullman High-Level Vision: Object Recognition and Visual Cognition , 1996 .

[28]  M. Maldonado,et al.  Bathymetric patterns of sponge distribution on the Bahamian slope , 1996 .

[29]  David L. Sheinberg,et al.  Visual object recognition. , 1996, Annual review of neuroscience.

[30]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[31]  P. Paatero Least squares formulation of robust non-negative factor analysis , 1997 .

[32]  T. Reincke,et al.  Silica uptake kinetics of Halichondria panicea in Kiel Bight , 1997 .

[33]  Susan T. Dumais,et al.  The latent semantic analysis theory of knowledge , 1997 .

[34]  Fernando Pereira,et al.  Aggregate and mixed-order Markov models for statistical language processing , 1997, EMNLP.

[35]  Peter Földiák,et al.  Sparse coding in the primate cortex , 1998 .

[36]  Marian Stewart Bartlett,et al.  Independent component representations for face recognition , 1998, Electronic Imaging.

[37]  M. Maldonado,et al.  An experimental approach to the ecological significance of microhabitat-scale movement in an encrusting sponge , 1999 .