High dimensional data that lies on or near a low dimensional manifold can be described by a collection of local linear models. Such a description, however, does not provide a global parameterization of the manifold—arguably an important goal of unsupervised learning. In this paper, we show how to learn a collection of local linear models that solves this more difficult problem. Our local linear models are represented by a mixture of factor analyzers, and the "global coordination" of these models is achieved by adding a regularizing term to the standard maximum likelihood objective function. The regularizer breaks a degeneracy in the mixture model's parameter space, favoring models whose internal coordinate systems are aligned in a consistent way. As a result, the internal coordinates change smoothly and continuously as one traverses a connected path on the manifold—even when the path crosses the domains of many different local models. The regularizer takes the form of a Kullback-Leibler divergence and illustrates an unexpected application of variational methods: not to perform approximate inference in intractable probabilistic models, but to learn more useful internal representations in tractable ones.
[1]
Garrison W. Cottrell,et al.
Non-Linear Dimensionality Reduction
,
1992,
NIPS.
[2]
Stephen M. Omohundro,et al.
Nonlinear Image Interpolation using Manifold Learning
,
1994,
NIPS.
[3]
Geoffrey E. Hinton,et al.
The EM algorithm for mixtures of factor analyzers
,
1996
.
[4]
Tomaso Poggio,et al.
Image Representations for Visual Learning
,
1996,
Science.
[5]
Geoffrey E. Hinton,et al.
Modeling the manifolds of images of handwritten digits
,
1997,
IEEE Trans. Neural Networks.
[6]
Nanda Kambhatla,et al.
Dimension Reduction by Local Principal Component Analysis
,
1997,
Neural Computation.
[7]
Christopher M. Bishop,et al.
GTM: The Generative Topographic Mapping
,
1998,
Neural Computation.
[8]
J. Tenenbaum,et al.
A global geometric framework for nonlinear dimensionality reduction.
,
2000,
Science.
[9]
S T Roweis,et al.
Nonlinear dimensionality reduction by locally linear embedding.
,
2000,
Science.