A Bernoulli relational model for nonlinear embedding

The notion of relations is extremely important in mathematics. In this paper, we use relations to describe the embedding problem and propose a novel stochastic relational model for nonlinear embedding. Given some relation among points in a high-dimensional space, we start from preserving the same relation in a low embedded space and model the relation as probabilistic distributions over these two spaces, respectively. We illustrate that the stochastic neighbor embedding and the Gaussian process latent variable model can be derived from our relational model. Moreover we devise a new stochastic embedding model and refer to it as Bernoulli relational embedding (BRE). BRE's ability in nonlinear dimensionality reduction is illustrated on a set of synthetic data and collections of bitmaps of handwritten digits and face images.

[1]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[2]  Geoffrey E. Hinton,et al.  Multiple Relational Embedding , 2004, NIPS.

[3]  Ian T. Nabney,et al.  Netlab: Algorithms for Pattern Recognition , 2002 .

[4]  Geoffrey E. Hinton,et al.  Neighbourhood Components Analysis , 2004, NIPS.

[5]  Bernard Kolman,et al.  Discrete Mathematical Structures , 1984 .

[6]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[7]  Mikhail Belkin,et al.  Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering , 2001, NIPS.

[8]  Geoffrey E. Hinton,et al.  Stochastic Neighbor Embedding , 2002, NIPS.

[9]  Fan Chung,et al.  Spectral Graph Theory , 1996 .

[10]  Yoram Singer,et al.  Multiclass Learning by Probabilistic Embeddings , 2002, NIPS.

[11]  Geoffrey E. Hinton,et al.  Global Coordination of Local Linear Models , 2001, NIPS.

[12]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[13]  Michael I. Jordan,et al.  Distance Metric Learning with Application to Clustering with Side-Information , 2002, NIPS.

[14]  Neil D. Lawrence,et al.  Gaussian Process Latent Variable Models for Visualisation of High Dimensional Data , 2003, NIPS.

[15]  Kun Huang,et al.  A unifying theorem for spectral embedding and clustering , 2003, AISTATS.

[16]  Matthew Brand,et al.  Continuous nonlinear dimensionality reduction by kernel Eigenmaps , 2003, IJCAI.