Connecting the Out-of-Sample and Pre-Image Problems in Kernel Methods

Kernel methods have been widely studied in the field of pattern recognition. These methods implicitly map, "the kernel trick," the data into a space which is more appropriate for analysis. Many manifold learning and dimensionality reduction techniques are simply kernel methods for which the mapping is explicitly computed. In such cases, two problems related with the mapping arise: The out-of-sample extension and the pre-image computation. In this paper we propose a new pre-image method based on the Nystrom formulation for the out-of-sample extension, showing the connections between both problems. We also address the importance of normalization in the feature space, which has been ignored by standard pre-image algorithms. As an example, we apply these ideas to the Gaussian kernel, and relate our approach to other popular pre-image methods. Finally, we show the application of these techniques in the study of dynamic shapes.

[1]  A. Atiya,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2005, IEEE Transactions on Neural Networks.

[2]  Nicolas Le Roux,et al.  Learning Eigenfunctions Links Spectral Embedding and Kernel PCA , 2004, Neural Computation.

[3]  Ronen Basri,et al.  Comparing images under variable illumination , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[4]  Ivor W. Tsang,et al.  The pre-image problem in kernel methods , 2003, IEEE Transactions on Neural Networks.

[5]  Mikhail Belkin,et al.  Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering , 2001, NIPS.

[6]  D. Donoho,et al.  Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[7]  Trevor F. Cox,et al.  Metric multidimensional scaling , 2000 .

[9]  Steven W. Zucker,et al.  Diffusion Maps and Geometric Harmonics for Automatic Target Recognition (ATR). Volume 2. Appendices , 2007 .

[10]  Bernhard Schölkopf,et al.  A kernel view of the dimensionality reduction of manifolds , 2004, ICML.

[11]  Yogesh Rathi,et al.  Shape-Based Approach to Robust Image Segmentation using Kernel PCA , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[12]  Roman Goldenberg,et al.  Behavior classification by eigendecomposition of periodic motions , 2005, Pattern Recognit..

[13]  Gunnar Rätsch,et al.  Kernel PCA and De-Noising in Feature Spaces , 1998, NIPS.

[14]  Roman Goldenberg,et al.  'Dynamism of a Dog on a Leash' or Behavior Classification by Eigen-Decomposition of Periodic Motions , 2002, ECCV.

[15]  J. Gower.  Adding a point to vector diagrams in multivariate analysis , 1968 .

[16]  Daniel Cremers,et al.  Nonlinear Shape Statistics via Kernel Spaces , 2001, DAGM-Symposium.

[17]  Kilian Q. Weinberger,et al.  Learning a kernel matrix for nonlinear dimensionality reduction , 2004, ICML.

[18]  David J. Fleet,et al.  3D People Tracking with Gaussian Process Dynamical Models , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[19]  Ahmed M. Elgammal,et al.  Gait tracking and recognition using person-dependent dynamic shape model , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[20]  Ronald R. Coifman,et al.  Data Fusion and Multicue Data Matching by Diffusion Maps , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[22]  Anuj Srivastava,et al.  Cyclostationary Processes on Shape Spaces for Gait-Based Recognition , 2006, ECCV.

[23]  Ron Kimmel,et al.  Representation Analysis and Synthesis of Lip Images Using Dimensionality Reduction , 2006, International Journal of Computer Vision.

[24]  Daniel Cremers,et al.  Dynamical statistical shape priors for level set-based tracking , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Bernhard Schölkopf,et al.  Kernel Principal Component Analysis , 1997, ICANN.

[26]  Guillermo Sapiro,et al.  Dynamic Shapes Average , 2003 .

[27]  Allen Tannenbaum,et al.  Statistical shape analysis using kernel PCA , 2006, Electronic Imaging.

[28]  Nicolas Le Roux,et al.  Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering , 2003, NIPS.

[29]  Namrata Vaswani,et al.  IEEE TRANSACTIONS ON IMAGE PROCESSING 1 A Generic Framework for Tracking using Particle Filter with Dynamic Shape Prior , 2022 .

[30]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[31]  Bernhard Schölkopf,et al.  Learning with kernels , 2001 .