Manifold Parzen Windows

The similarity between objects is a fundamental element of many learning algorithms. Most non-parametric methods take this similarity to be fixed, but much recent work has shown the advantages of learning it, in particular to exploit the local invariances in the data or to capture the possibly non-linear manifold on which most of the data lies. We propose a new non-parametric kernel density estimation method which captures the local structure of an underlying manifold through the leading eigenvectors of regularized local covariance matrices. Experiments in density estimation show significant improvements with respect to Parzen density estimators. The density estimators can also be used within Bayes classifiers, yielding classification rates similar to SVMs and much superior to the Parzen classifier.

[1]  E. Parzen On Estimation of a Probability Density Function and Mode , 1962 .

[2]  Keinosuke Fukunaga,et al.  The optimal distance measure for nearest neighbor classification , 1981, IEEE Trans. Inf. Theory.

[3]  David J. Hand,et al.  The multi-class metric problem in nearest neighbour discrimination rules , 1990, Pattern Recognit..

[4]  A. Izenman Recent Developments in Nonparametric Density Estimation , 1991 .

[5]  Jerome H. Friedman,et al.  Flexible Metric Nearest Neighbor Classification , 1994 .

[6]  Geoffrey E. Hinton,et al.  Recognizing Handwritten Digits Using Mixtures of Linear Models , 1994, NIPS.

[7]  Robert Tibshirani,et al.  Discriminant Adaptive Nearest Neighbor Classification and Regression , 1995, NIPS.

[8]  Geoffrey E. Hinton,et al.  The EM algorithm for mixtures of factor analyzers , 1996 .

[9]  Yann LeCun,et al.  Transformation Invariance in Pattern Recognition-Tangent Distance and Tangent Propagation , 1996, Neural Networks: Tricks of the Trade.

[10]  Michael E. Tipping,et al.  Mixtures of Principal Component Analysers , 1997 .

[11]  Christopher M. Bishop,et al.  Mixtures of Probabilistic Principal Component Analyzers , 1999, Neural Computation.

[12]  Zoubin Ghahramani,et al.  Variational Inference for Bayesian Mixtures of Factor Analysers , 1999, NIPS.

[13]  Hermann Ney,et al.  Structured Covariance Matrices for Statistical Image Object Recognition , 2000, DAGM-Symposium.

[14]  Hermann Ney,et al.  A Probabilistic View on Tangent Distance , 2000, DAGM-Symposium.

[15]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[16]  Pascal Vincent,et al.  K-Local Hyperplane and Convex Distance Nearest Neighbor Algorithms , 2001, NIPS.

[17]  Yee Whye Teh,et al.  Automatic Alignment of Local Representations , 2002, NIPS.

[18]  Matthew Brand,et al.  Charting a Manifold , 2002, NIPS.

[19]  Joshua B. Tenenbaum,et al.  Global Versus Local Methods in Nonlinear Dimensionality Reduction , 2002, NIPS.