Learning a Metric Space for Neighbourhood Topology Estimation: Application to Manifold Learning

Manifold learning algorithms rely on a neighbourhood graph to provide an estimate of the data’s local topology. Unfortunately, current methods for estimating local topology assume local Euclidean geometry and locally uniform data density, which often leads to poor data embeddings. We address these shortcomings by proposing a framework that combines local learning with parametric density estimation for local topology estimation. Given a data set D X , we rst estimate a new metric space ( X;dX) that characterizes the varying sample density ofX in X, then use (X;dX) as a new (pilot) input space for the graph construction step of the manifold learning process. The proposed framework results in signicantly improved embeddings, which we demonstrated objectively by assessing clustering accuracy.

[1]  Gregory Piatetsky-Shapiro,et al.  High-Dimensional Data Analysis: The Curses and Blessings of Dimensionality , 2000 .

[2]  Frank P. Ferrie,et al.  Relaxed Exponential Kernels for Unsupervised Learning , 2011, DAGM-Symposium.

[3]  C. R. Rao,et al.  Information and the Accuracy Attainable in the Estimation of Statistical Parameters , 1992 .

[4]  Charles Elkan,et al.  Using the Triangle Inequality to Accelerate k-Means , 2003, ICML.

[5]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[6]  T. Kailath The Divergence and Bhattacharyya Distance Measures in Signal Selection , 1967 .

[7]  Kilian Q. Weinberger,et al.  Unsupervised Learning of Image Manifolds by Semidefinite Programming , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[8]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[9]  Elad Hazan,et al.  Sparse Approximate Solutions to Semidefinite Programs , 2008, LATIN.

[10]  Geoffrey E. Hinton,et al.  Stochastic Neighbor Embedding , 2002, NIPS.

[11]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[12]  Avleen Singh Bijral,et al.  Semi-supervised Learning with Density Based Distances , 2011, UAI.

[13]  Tony R. Martinez,et al.  Robust manifold learning with CycleCut , 2012, Connect. Sci..

[14]  Léon Bottou,et al.  Local Learning Algorithms , 1992, Neural Computation.

[15]  Tony Jebara,et al.  Structure preserving embedding , 2009, ICML '09.

[16]  Matthew Brand,et al.  Charting a Manifold , 2002, NIPS.

[17]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[18]  Pascal Vincent,et al.  Manifold Parzen Windows , 2002, NIPS.

[19]  Christopher K. I. Williams On a Connection between Kernel PCA and Metric Multidimensional Scaling , 2004, Machine Learning.

[20]  Frank P. Ferrie,et al.  A Note on Metric Properties for Some Divergence Measures: The Gaussian Case , 2012, ACML.

[21]  Miguel Á. Carreira-Perpiñán,et al.  Proximity Graphs for Clustering and Manifold Learning , 2004, NIPS.

[22]  Michael I. Jordan,et al.  Distance Metric Learning with Application to Clustering with Side-Information , 2002, NIPS.

[23]  Nicolas Le Roux,et al.  Learning Eigenfunctions Links Spectral Embedding and Kernel PCA , 2004, Neural Computation.

[24]  Ronald L. Rivest,et al.  Introduction to Algorithms, Second Edition , 2001 .

[25]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[26]  I. J. Schoenberg,et al.  Metric spaces and positive definite functions , 1938 .

[27]  Trevor Darrell,et al.  Nearest-Neighbor Methods in Learning and Vision: Theory and Practice (Neural Information Processing) , 2006 .

[28]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[29]  Tim W. Nattkemper,et al.  ISOLLE: LLE with geodesic distance , 2006, Neurocomputing.

[30]  A. Householder,et al.  Discussion of a set of points in terms of their mutual distances , 1938 .

[31]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[32]  Yaoliang Yu,et al.  Regularizers versus Losses for Nonlinear Dimensionality Reduction: A Factored View with New Convex Relaxations , 2012, ICML.

[33]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[34]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[35]  I. Johnstone On the distribution of the largest eigenvalue in principal components analysis , 2001 .

[36]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[37]  Mukund Balasubramanian,et al.  The Isomap Algorithm and Topological Stability , 2002, Science.