Manifold Structured Prediction

Structured prediction provides a general framework to deal with supervised problems where the outputs have semantically rich structure. While classical approaches consider finite, albeit potentially huge, output spaces, in this paper we discuss how structured prediction can be extended to a continuous scenario. Specifically, we study a structured prediction approach to manifold-valued regression. We characterize a class of problems for which the considered approach is statistically consistent and study how geometric optimization can be used to compute the corresponding estimator. Promising experimental results on both simulated and real data complete our study.

[1]  Søren Hauberg,et al.  A Geometric take on Metric Learning , 2012, NIPS.

[2]  Florian Steinke,et al.  Non-parametric Regression Between Manifolds , 2008, NIPS.

[3]  Benjamin Recht,et al.  Random Features for Large-Scale Kernel Machines , 2007, NIPS.

[4]  Alexander J. Smola,et al.  Nonparametric Quantile Estimation , 2006, J. Mach. Learn. Res..

[5]  Lorenzo Rosasco,et al.  FALKON: An Optimal Large Scale Kernel Method , 2017, NIPS.

[6]  Maher Moakher,et al.  Symmetric Positive-Definite Matrices: From Geometry to Applications and Visualization , 2006, Visualization and Processing of Tensor Fields.

[7]  Alessandro Rudi,et al.  Differential Properties of Sinkhorn Approximation for Learning with Wasserstein Distance , 2018, NeurIPS.

[8]  Claude Sammut,et al.  Classification of Multivariate Time Series and Structured Data Using Constructive Induction , 2005, Machine Learning.

[9]  Daniel Marcu,et al.  Practical structured learning techniques for natural language processing , 2006 .

[10]  P. Thomas Fletcher,et al.  Geodesic Regression and the Theory of Least Squares on Riemannian Manifolds , 2012, International Journal of Computer Vision.

[11]  B. Fishel,et al.  TOPOLOGICAL VECTOR SPACES, DISTRIBUTIONS AND KERNELS , 1969 .

[12]  A. Caponnetto,et al.  Optimal Rates for the Regularized Least-Squares Algorithm , 2007, Found. Comput. Math..

[13]  Suvrit Sra,et al.  First-order Methods for Geodesically Convex Optimization , 2016, COLT.

[14]  Jeffrey S. Morris.  Functional Regression , 2014, 1406.4068.

[15]  Maxime Sangnier,et al.  Output Fisher embedding regression , 2018, Machine Learning.

[16]  Thomas Hofmann,et al.  Predicting Structured Data (Neural Information Processing) , 2007 .

[17]  Alessandro Rudi,et al.  Sharp Analysis of Learning with Discrete Losses , 2019, AISTATS.

[18]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[19]  Florence d'Alché-Buc,et al.  A Structured Prediction Approach for Label Ranking , 2018, NeurIPS.

[20]  Thanh Tran,et al.  Relational Kernel Machines for Learning from Graph-Structured RDF Data , 2011, ESWC.

[21]  C. Villani.  Optimal Transport: Old and New , 2008 .

[22]  Lorenzo Rosasco,et al.  Less is More: Nyström Computational Regularization , 2015, NIPS.

[23]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2003, ICTAI.

[24]  Bernhard Schölkopf,et al.  Nonparametric Regression between General Riemannian Manifolds , 2010, SIAM J. Imaging Sci..

[25]  Sebastian Nowozin,et al.  Structured Learning and Prediction in Computer Vision , 2011, Found. Trends Comput. Graph. Vis..

[26]  Lorenzo Rosasco,et al.  Learning with SGD and Random Features , 2018, NeurIPS.

[27]  Gökhan BakIr,et al.  Predicting Structured Data , 2008 .

[28]  P. Thomas Fletcher,et al.  Polynomial Regression on Riemannian Manifolds , 2012, ECCV.

[29]  I. Holopainen.  Riemannian Geometry , 1927, Nature.

[30]  Barbara Hammer,et al.  Time Series Prediction for Graphs in Kernel and Dissimilarity Spaces , 2017, Neural Processing Letters.

[31]  Bernhard Schölkopf,et al.  Sparse Greedy Matrix Approximation for Machine Learning , 2000, International Conference on Machine Learning.

[32]  Janez Mrcun,et al.  On isomorphisms of algebras of smooth functions , 2003, math/0309179.

[33]  Volkan Cevher,et al.  Optimal rates for spectral algorithms with least-squares regression over Hilbert spaces , 2018, Applied and Computational Harmonic Analysis.

[34]  Lorenzo Rosasco,et al.  Consistent Multitask Learning with Nonlinear Output Relations , 2017, NIPS.

[35]  Lorenzo Rosasco,et al.  A Consistent Regularization Approach for Structured Prediction , 2016, NIPS.

[36]  Neil D. Lawrence,et al.  Kernels for Vector-Valued Functions: a Review , 2011, Found. Trends Mach. Learn..

[37]  Lorenzo Rosasco,et al.  Generalization Properties of Learning with Random Features , 2016, NIPS.

[38]  Shun-ichi Amari,et al.  Methods of information geometry , 2000 .

[39]  Douwe Kiela,et al.  Poincaré Embeddings for Learning Hierarchical Representations , 2017, NIPS.

[40]  Frank Nielsen,et al.  Clustering in Hilbert simplex geometry , 2017, Geometric Structures of Information.

[41]  Angela Spalsbury,et al.  The Joys of Haar Measure , 2014 .

[42]  Alessandro Rudi,et al.  Localized Structured Prediction , 2018, NeurIPS.

[43]  Andreas Christmann,et al.  Support vector machines , 2008, Data Mining and Knowledge Discovery Handbook.

[44]  Levent Tunçel,et al.  Optimization algorithms on matrix manifolds , 2009, Math. Comput..

[45]  Emmanuel Hebey.  Nonlinear analysis on manifolds: Sobolev spaces and inequalities , 1999 .

[46]  Franz-Erich Wolter.  Distance function and cut loci on a complete Riemannian manifold , 1979 .

[47]  Don R. Hush,et al.  Optimal Rates for Regularized Least Squares Regression , 2009, COLT.

[48]  Charles E. Heckler,et al.  Applied Multivariate Statistical Analysis , 2005, Technometrics.

[49]  Silvere Bonnabel,et al.  Stochastic Gradient Descent on Riemannian Manifolds , 2011, IEEE Transactions on Automatic Control.

[50]  Charles A. Micchelli,et al.  On Learning Vector-Valued Functions , 2005, Neural Computation.

[51]  Michael I. Jordan,et al.  On the Consistency of Ranking Algorithms , 2010, ICML.

[52]  Lorenzo Rosasco,et al.  NYTRO: When Subsampling Meets Early Stopping , 2015, AISTATS.

[53]  R. Bhatia.  Positive Definite Matrices , 2007 .

[54]  Douwe Kiela,et al.  Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry , 2018, ICML.

[55]  Grigorios Tsoumakas,et al.  Mining Multi-label Data , 2010, Data Mining and Knowledge Discovery Handbook.