An RSVM based two-teachers-one-student semi-supervised learning algorithm

Based on the reduced SVM, we propose a multi-view algorithm, two-teachers-one-student, for semi-supervised learning. With RSVM, different from typical multi-view methods, reduced sets suggest different views in the represented kernel feature space rather than in the input space. No label information is necessary when we select reduced sets, and this makes applying RSVM to SSL possible. Our algorithm blends the concepts of co-training and consensus training. Through co-training, the classifiers generated by two views can "teach" the third classifier from the remaining view to learn, and this process is performed for each choice of teachers-student combination. By consensus training, predictions from more than one view can give us higher confidence for labeling unlabeled data. The results show that the proposed 2T1S achieves high cross-validation accuracy, even compared to the training with all the label information available.

[1]  J. A. Hartigan,et al.  A k-means clustering algorithm , 1979 .

[2]  Jason Weston,et al.  Large Scale Transductive SVMs , 2006, J. Mach. Learn. Res..

[3]  Xiaojin Zhu,et al.  --1 CONTENTS , 2006 .

[4]  Ethem Alpaydin,et al.  Introduction to machine learning , 2004, Adaptive computation and machine learning.

[5]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[6]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[7]  C. Oliveira.  Splitting the Unsupervised and Supervised Components of Semi-Supervised Learning , 2005 .

[8]  Maurice K. Wong,et al.  Algorithm AS136: A k-means clustering algorithm. , 1979 .

[9]  Alexander Zien,et al.  Semi-Supervised Learning , 2006 .

[10]  Linda Kaufman,et al.  Solving the quadratic programming problem arising in support vector classification , 1999 .

[11]  Bernhard Schölkopf,et al.  Sparse Greedy Matrix Approximation for Machine Learning , 2000, International Conference on Machine Learning.

[12]  Su-Yun Huang,et al.  Model selection for support vector machines via uniform design , 2007, Comput. Stat. Data Anal..

[13]  Tin Kam Ho,et al.  Building projectable classifiers of arbitrary complexity , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[14]  D. J. Newman,et al.  UCI Repository of Machine Learning Database , 1998 .

[15]  Mikhail Belkin,et al.  Manifold Regularization : A Geometric Framework for Learning from Examples , 2004 .

[16]  Bir Bhanu,et al.  Active concept learning in image databases , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[17]  Aristidis Likas,et al.  Semi-supervised and active learning with the probabilistic RBF classifier , 2008, Neurocomputing.

[18]  Su-Yun Huang,et al.  Incremental Reduced Support Vector Machines , 2001 .

[19]  Mikhail Belkin,et al.  Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples , 2006, J. Mach. Learn. Res..

[20]  Haitao Zhao.  Combining labeled and unlabeled data with graph embedding , 2006, Neurocomputing.

[21]  Su-Yun Huang,et al.  Reduced Support Vector Machines: A Statistical Theory , 2007, IEEE Transactions on Neural Networks.

[22]  Avrim Blum,et al.  The Bottleneck , 2021, Monopsony Capitalism.

[23]  Olvi L. Mangasarian,et al.  Nonlinear Programming , 1969 .

[24]  Zhi-Hua Zhou,et al.  Tri-training: exploiting unlabeled data using three classifiers , 2005, IEEE Transactions on Knowledge and Data Engineering.

[25]  Ayhan Demiriz,et al.  Semi-Supervised Support Vector Machines , 1998, NIPS.

[26]  Matthias W. Seeger,et al.  Using the Nyström Method to Speed Up Kernel Machines , 2000, NIPS.

[27]  Ronald Rosenfeld,et al.  Semi-supervised learning with graphs , 2005 .

[28]  Günther Palm,et al.  Semi-supervised learning for tree-structured ensembles of RBF networks with Co-Training , 2010, Neural Networks.

[29]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[30]  Yuh-Jye Lee,et al.  RSVM: Reduced Support Vector Machines , 2001, SDM.

[31]  Yuh-Jye Lee,et al.  SSVM: A Smooth Support Vector Machine for Classification , 2001, Comput. Optim. Appl..

[32]  Avrim Blum,et al.  Learning from Labeled and Unlabeled Data using Graph Mincuts , 2001, ICML.

[33]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[34]  Yuh-Jye Lee,et al.  Variant Methods of Reduced Set Selection for Reduced Support Vector Machines , 2010, J. Inf. Sci. Eng..

[35]  Fei Wang,et al.  Robust self-tuning semi-supervised learning , 2007, Neurocomputing.