Locality Preserving Projections

Many problems in information processing involve some form of dimensionality reduction. In this paper, we introduce Locality Preserving Projections (LPP). These are linear projective maps that arise by solving a variational problem that optimally preserves the neighborhood structure of the data set. LPP should be seen as an alternative to Principal Component Analysis (PCA) – a classical linear technique that projects the data along the directions of maximal variance. When the high dimensional data lies on a low dimensional manifold embedded in the ambient space, the Locality Preserving Projections are obtained by finding the optimal linear approximations to the eigenfunctions of the Laplace Beltrami operator on the manifold. As a result, LPP shares many of the data representation properties of nonlinear techniques such as Laplacian Eigenmaps or Locally Linear Embedding. Yet LPP is linear and more crucially is defined everywhere in ambient space rather than just on the training data points. This is borne out by illustrative examples on some high dimensional data sets.

[1]  Matthew Brand,et al.  Charting a Manifold , 2002, NIPS.

[2]  Ming-Hsuan Yang,et al.  Kernel Eigenfaces vs. Kernel Fisherfaces: Face recognition using kernel methods , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[3]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[4]  Jieping Ye,et al.  Two-Dimensional Linear Discriminant Analysis , 2004, NIPS.

[5]  Gary L. Miller,et al.  Graph Embeddings and Laplacian Eigenvalues , 2000, SIAM J. Matrix Anal. Appl..

[6]  Changbo Hu,et al.  Manifold of facial expression , 2003, 2003 IEEE International SOI Conference. Proceedings (Cat. No.03CH37443).

[7]  Yuxiao Hu,et al.  Face recognition using Laplacianfaces , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Richard A. Harshman,et al.  Indexing by Latent Semantic Analysis , 1990, J. Am. Soc. Inf. Sci..

[9]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[10]  Amnon Shashua,et al.  Manifold pursuit: a new approach to appearance based recognition , 2002, Object recognition supported by user interaction for service robots.

[11]  Takeshi Shakunaga,et al.  Decomposed eigenface for face recognition under various lighting conditions , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[12]  Shuicheng Yan,et al.  Ranking prior likelihood distributions for Bayesian shape localization framework , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[13]  Geoffrey E. Hinton,et al.  Global Coordination of Local Linear Models , 2001, NIPS.

[14]  F. Chung.  Spectral Graph Theory, Regional Conference Series in Math. , 1997 .

[15]  Andrew W. Fitzgibbon,et al.  Joint manifold distance: a new approach to appearance based clustering , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[16]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[17]  Jian Yang,et al.  Two-dimensional PCA: a new approach to appearance-based face representation and recognition , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  John M. Lee.  Introduction to Smooth Manifolds , 2002 .

[19]  Alex Pentland,et al.  Probabilistic Visual Learning for Object Representation , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Jon Louis Bentley,et al.  Multidimensional divide-and-conquer , 1980, CACM.

[21]  Sayan Mukherjee,et al.  Feature Selection for SVMs , 2000, NIPS.

[22]  Paul A. Viola,et al.  Restructuring Sparse High Dimensional Data for Effective Retrieval , 1998, NIPS.

[23]  Xin Liu,et al.  Document clustering based on non-negative matrix factorization , 2003, SIGIR.

[24]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[25]  Jakob Nielsen,et al.  Automating the assignment of submitted manuscripts to reviewers , 1992, SIGIR '92.

[26]  L Sirovich,et al.  Low-dimensional procedure for the characterization of human faces. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[27]  Susan T. Dumais,et al.  Personalized information delivery: an analysis of information filtering methods , 1992, CACM.

[28]  Garrison W. Cottrell,et al.  Latent semantic indexing is an optimal special case of multidimensional scaling , 1992, SIGIR '92.

[29]  Rie Kubota Ando.  Latent semantic space: iterative scaling improves precision of inter-document similarity measurement , 2000, SIGIR '00.

[30]  J. S. Urban Hjorth,et al.  Computer Intensive Statistical Methods: Validation, Model Selection, and Bootstrap , 1993 .

[31]  Volker Roth,et al.  Feature Selection in Clustering Problems , 2003, NIPS.

[32]  G KoldaTamara,et al.  A semidiscrete matrix decomposition for latent semantic indexing information retrieval , 1998 .

[33]  David Cohn,et al.  Informed Projections , 2002, NIPS.

[34]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[35]  G. Baudat,et al.  Generalized Discriminant Analysis Using a Kernel Approach , 2000, Neural Computation.

[36]  Mikhail Belkin,et al.  Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering , 2001, NIPS.

[37]  David J. Kriegman,et al.  Video-based face recognition using probabilistic appearance manifolds , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[38]  Terence Sim,et al.  The CMU Pose, Illumination, and Expression Database , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[39]  P. Jonathon Phillips,et al.  Support Vector Machines Applied to Face Recognition , 1998, NIPS.

[40]  Rong Xiao,et al.  Boosting chain learning for object detection , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[41]  David J. Kriegman,et al.  Clustering appearances of objects under varying illumination conditions , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[42]  Sanjoy Dasgupta,et al.  Experiments with Random Projection , 2000, UAI.

[43]  David J. Kriegman,et al.  Image Clustering with Metric, Local Linear Structure, and Affine Symmetry , 2004, ECCV.

[44]  Avinash C. Kak,et al.  PCA versus LDA , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  Fan Chung,et al.  Spectral Graph Theory , 1996 .

[46]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[47]  Nuno Vasconcelos,et al.  Feature Selection by Maximum Marginal Diversity , 2002, NIPS.

[48]  Hiroshi Murase,et al.  Visual learning and recognition of 3-d objects from appearance , 2005, International Journal of Computer Vision.

[49]  Hongyuan Zha,et al.  Isometric Embedding and Continuum ISOMAP , 2003, ICML.

[50]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[51]  Wei-Ying Ma,et al.  Locality preserving clustering for image database , 2004, MULTIMEDIA '04.

[52]  Santosh S. Vempala,et al.  Latent semantic indexing: a probabilistic analysis , 1998, PODS '98.

[53]  H. White,et al.  Cross-Validation Estimates IMSE , 1993, NIPS 1993.

[54]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[55]  Jon M. Kleinberg,et al.  Two algorithms for nearest-neighbor search in high dimensions , 1997, STOC '97.

[56]  Lawrence K. Saul,et al.  Think Globally, Fit Locally: Unsupervised Learning of Low Dimensional Manifold , 2003, J. Mach. Learn. Res..

[57]  C. Ding.  A similarity-based probability model for latent semantic indexing , 1999, SIGIR '99.

[58]  Yves Grandvalet,et al.  Adaptive Scaling for Feature Selection in SVMs , 2002, NIPS.

[59]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[60]  R. Chellappa,et al.  Subspace Linear Discriminant Analysis for Face Recognition , 1999 .

[61]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[62]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[63]  M. Chavance.  [Jackknife and bootstrap]. , 1992, Revue d'epidemiologie et de sante publique.

[64]  Amnon Shashua,et al.  Principal Component Analysis over Continuous Subspaces and Intersection of Half-Spaces , 2002, ECCV.

[65]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[66]  Norbert Krüger,et al.  Face recognition by elastic bunch graph matching , 1997, Proceedings of International Conference on Image Processing.

[67]  H. Sebastian Seung,et al.  The Manifold Ways of Perception , 2000, Science.

[68]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[69]  Demetri Terzopoulos,et al.  Multilinear subspace analysis of image ensembles , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[70]  Yair Weiss,et al.  Segmentation using eigenvectors: a unifying view , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[71]  Nicolas Le Roux,et al.  Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering , 2003, NIPS.

[72]  Jie Yang,et al.  An Efficient LDA Algorithm for Face Recognition , 2000 .

[73]  Lillian Lee,et al.  Iterative Residual Rescaling: An Analysis and Generalization of LSI , 2001, SIGIR 2002.

[74]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[75]  B. Mohar.  Some applications of Laplace eigenvalues of graphs , 1997 .

[76]  Heikki Mannila,et al.  Random projection in dimensionality reduction: applications to image and text data , 2001, KDD '01.