Two Manifold Problems with Applications to Nonlinear System Identification

Recently, there has been much interest in spectral approaches to learning manifolds-- so-called kernel eigenmap methods. These methods have had some successes, but their applicability is limited because they are not robust to noise. To address this limitation, we look at two-manifold problems, in which we simultaneously reconstruct two related manifolds, each representing a different view of the same data. By solving these interconnected learning problems together, two-manifold algorithms are able to succeed where a non-integrated approach would fail: each view allows us to suppress noise in the other, reducing bias. We propose a class of algorithms for two-manifold problems, based on spectral decomposition of cross-covariance operators in Hilbert space, and discuss when two-manifold problems are useful. Finally, we demonstrate that solving a two-manifold problem can aid in learning a nonlinear dynamical system from limited data.

[1]  G. Reinsel,et al.  Multivariate Reduced-Rank Regression: Theory and Applications , 1998 .

[2]  Neil D. Lawrence,et al.  Spectral Dimensionality Reduction via Maximum Entropy , 2011, AISTATS.

[3]  Ker-Chau Li,et al.  Sliced Inverse Regression for Dimension Reduction , 1991 .

[4]  Le Song,et al.  A Hilbert Space Embedding for Distributions , 2007, Discovery Science.

[5]  Nuno Vasconcelos,et al.  Maximum Covariance Unfolding : Manifold Learning for Bimodal Data , 2011, NIPS.

[6]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[7]  Michael I. Jordan,et al.  Dimensionality Reduction for Supervised Learning with Reproducing Kernel Hilbert Spaces , 2004, J. Mach. Learn. Res..

[8]  Byron Boots,et al.  Predictive State Temporal Difference Learning , 2010, NIPS.

[9]  R. Cook,et al.  Theory & Methods: Special Invited Paper: Dimension Reduction and Visualization in Discriminant Analysis (with discussion) , 2001 .

[10]  Ian T. Jolliffe,et al.  Principal Component Analysis , 2002, International Encyclopedia of Statistical Science.

[11]  Daniel D. Lee,et al.  Semisupervised alignment of manifolds , 2005, AISTATS.

[12]  Steven J. Bradtke,et al.  Linear Least-Squares algorithms for temporal difference learning , 2004, Machine Learning.

[13]  Sham M. Kakade,et al.  A spectral algorithm for learning Hidden Markov Models , 2008, J. Comput. Syst. Sci..

[14]  Byron Boots,et al.  An Online Spectral Learning Algorithm for Partially Observable Nonlinear Dynamical Systems , 2011, AAAI.

[15]  Bart De Moor,et al.  Subspace Identification for Linear Systems: Theory ― Implementation ― Applications , 2011 .

[16]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[17]  Kenji Fukumizu,et al.  Consistency of Kernel Canonical Correlation Analysis , 2005 .

[18]  Bernhard Schölkopf,et al.  A kernel view of the dimensionality reduction of manifolds , 2004, ICML.

[19]  Le Song,et al.  Hilbert Space Embeddings of Hidden Markov Models , 2010, ICML.

[20]  Byron Boots,et al.  Reduced-Rank Hidden Markov Models , 2009, AISTATS.

[21]  H. Hotelling The most predictable criterion. , 1935 .

[22]  Chang Wang,et al.  A General Framework for Manifold Alignment , 2009, AAAI Fall Symposium: Manifold Learning and Its Applications.

[23]  J. Pearl Causality: Models, Reasoning and Inference , 2000 .

[24]  Michael I. Jordan,et al.  Regression on manifolds using kernel dimension reduction , 2007, ICML '07.

[25]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[26]  Jianping Yin,et al.  Robust local tangent space alignment via iterative weighted PCA , 2011, Neurocomputing.

[27]  Deyi Li,et al.  Neighborhood smoothing embedding for noisy manifold learning , 2008, 2008 IEEE International Conference on Granular Computing.

[28]  Kilian Q. Weinberger,et al.  Learning a kernel matrix for nonlinear dimensionality reduction , 2004, ICML.

[29]  Byron Boots,et al.  Closing the learning-planning loop with predictive state representations , 2009, Int. J. Robotics Res..

[30]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[31]  Jianping Yin,et al.  Robust Local Tangent Space Alignment , 2009, ICONIP.