Fast and Provably Effective Multi-view Classification with Landmark-Based SVM

We introduce a fast and theoretically founded method for learning landmark-based SVMs in a multi-view classification setting which leverages the complementary information of the different views and linearly scales with the size of the dataset. The proposed method – called MVL-SVM – applies a non-linear projection to the dataset through multi-view similarity estimates w.r.t. a small set of randomly selected landmarks, before learning a linear SVM in this latent space joining all the views. Using the uniform stability framework, we prove that our algorithm is robust to slight changes in the training set leading to a generalization bound depending on the number of views and landmarks. We also show that our method can be easily adapted to a missing-view scenario by only reconstructing the similarities to the landmarks. Empirical results, both in complete and missing view settings, highlight the superior performances of our method, in terms of accuracy and execution time, w.r.t. state of the art techniques. Code related to this paper is available at: https://github.com/vzantedeschi/multiviewLSVM.

[1]  Vittorio Murino,et al.  A unifying framework for vector-valued manifold regularization and multi-view learning , 2013, ICML.

[2]  Shiliang Sun,et al.  Multi-view learning overview: Recent progress and new challenges , 2017, Inf. Fusion.

[3]  Barbara Engelhardt Martin.  Bayesian group factor analysis with structured sparsity , 2016 .

[4]  Vittorio Murino,et al.  A Unifying Framework in Vector-valued Reproducing Kernel Hilbert Spaces for Manifold Regularization and Co-Regularized Multi-view Learning , 2014, J. Mach. Learn. Res..

[5]  Juho Rousu,et al.  Multi-view kernel completion , 2016, Machine Learning.

[6]  Chih-Jen Lin,et al.  LIBLINEAR: A Library for Large Linear Classification , 2008, J. Mach. Learn. Res..

[7]  Patrick Gallinari,et al.  SGD-QN: Careful Quasi-Newton Stochastic Gradient Descent , 2009, J. Mach. Learn. Res..

[8]  André Elisseeff,et al.  Stability and Generalization , 2002, J. Mach. Learn. Res..

[9]  Matthias W. Seeger,et al.  Using the Nyström Method to Speed Up Kernel Machines , 2000, NIPS.

[10]  Hachem Kadri,et al.  Multi-view Metric Learning in Vector-valued Kernel Spaces , 2018, AISTATS.

[11]  Stéphane Ayache,et al.  The Multi-Task Learning View of Multimodal Data , 2013, ACML.

[12]  Shiliang Sun,et al.  A survey of multi-view machine learning , 2013, Neural Computing and Applications.

[13]  Charles A. Micchelli,et al.  On Learning Vector-Valued Functions , 2005, Neural Computation.

[14]  Jason Weston,et al.  Breaking SVM Complexity with Cross-Training , 2004, NIPS.

[15]  Shiliang Sun,et al.  Multi-view Laplacian Support Vector Machines , 2011, ADMA.

[16]  John Shawe-Taylor,et al.  Two view learning: SVM-2K, Theory and Practice , 2005, NIPS.

[17]  Piyush Rai,et al.  Multiview Clustering with Incomplete Views , 2010 .

[18]  Rong Jin,et al.  Distance Metric Learning: A Comprehensive Survey , 2006 .

[19]  Jingjing Tang,et al.  Multiview Privileged Support Vector Machines , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[20]  Philip H. S. Torr,et al.  Locally Linear Support Vector Machines , 2011, ICML.

[21]  Marc Sebban,et al.  A Survey on Metric Learning for Feature Vectors and Structured Data , 2013, ArXiv.

[22]  Massih-Reza Amini,et al.  PAC-Bayesian Analysis for a Two-Step Hierarchical Multiview Learning Approach , 2016, ECML/PKDD.

[23]  Avrim Blum,et al.  The Bottleneck , 2021, Monopsony Capitalism.

[24]  Ethem Alpaydin,et al.  Multiple Kernel Learning Algorithms , 2011, J. Mach. Learn. Res..

[25]  Ingo Steinwart,et al.  Sparseness of Support Vector Machines , 2003, J. Mach. Learn. Res..

[26]  Massih-Reza Amini,et al.  Learning from Multiple Partially Observed Views - an Application to Multilingual Text Categorization , 2009, NIPS.

[27]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[28]  Dacheng Tao,et al.  A Survey on Multi-view Learning , 2013, ArXiv.

[29]  Rémi Emonet,et al.  L3-SVMs: Landmarks-based Linear Local Support Vectors Machines , 2017, ArXiv.