Semisupervised Learning Using Bayesian Interpretation: Application to LS-SVM

Bayesian reasoning provides an ideal basis for representing and manipulating uncertain knowledge, with the result that many interesting algorithms in machine learning are based on Bayesian inference. In this paper, we use the Bayesian approach with one and two levels of inference to model the semisupervised learning problem and give its application to the successful kernel classifier support vector machine (SVM) and its variant least-squares SVM (LS-SVM). Taking advantage of Bayesian interpretation of LS-SVM, we develop a semisupervised learning algorithm for Bayesian LS-SVM using our approach based on two levels of inference. Experimental results on both artificial and real pattern recognition problems show the utility of our method.

[1]  Xiaojin Zhu,et al.  Semi-Supervised Learning Literature Survey , 2005 .

[2]  Carl Gold,et al.  Bayesian approach to feature selection and parameter tuning for support vector machine classifiers , 2005, Neural Networks.

[3]  Thorsten Joachims,et al.  Transductive Learning via Spectral Graph Partitioning , 2003, ICML.

[4]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[5]  Wei Chu,et al.  An improved conjugate gradient scheme to the solution of least squares SVM , 2005, IEEE Transactions on Neural Networks.

[6]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[7]  Alexander Zien,et al.  Semi-Supervised Learning , 2006 .

[8]  J. Marden,et al.  Hypothesis Testing: From p Values to Bayes Factors , 2000 .

[9]  Ellen Riloff,et al.  Learning subjective nouns using extraction pattern bootstrapping , 2003, CoNLL.

[10]  Amrish Patel,et al.  Semi-supervised classification using sparse Gaussian process regression , 2009, IJCAI 2009.

[11]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[12]  Avrim Blum,et al.  Learning from Labeled and Unlabeled Data using Graph Mincuts , 2001, ICML.

[13]  Licheng Jiao,et al.  Fast Sparse Approximation for Least Squares Support Vector Machine , 2007, IEEE Transactions on Neural Networks.

[14]  Dustin Boswell,et al.  Introduction to Support Vector Machines , 2002 .

[15]  Wei Chu,et al.  Semi-Supervised Gaussian Process Classifiers , 2007, IJCAI.

[16]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machines , 2002 .

[17]  Avrim Blum,et al.  The Bottleneck , 2021, Monopsony Capitalism.

[18]  Johan A. K. Suykens,et al.  Bayesian Framework for Least-Squares Support Vector Machine Classifiers, Gaussian Processes, and Kernel Fisher Discriminant Analysis , 2002, Neural Computation.

[19]  Peter Sollich,et al.  Probabilistic Methods for Support Vector Machines , 1999, NIPS.

[20]  Neil D. Lawrence,et al.  Semi-supervised Learning via Gaussian Processes , 2004, NIPS.

[21]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[22]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[23]  Zoubin Ghahramani,et al.  Semi-supervised learning : from Gaussian fields to Gaussian processes , 2003 .

[24]  Ayhan Demiriz,et al.  Semi-Supervised Support Vector Machines , 1998, NIPS.

[25]  K. Johana,et al.  Benchmarking Least Squares Support Vector Machine Classifiers , 2022 .

[26]  Theo J. A. de Vries,et al.  Pruning error minimization in least squares support vector machines , 2003, IEEE Trans. Neural Networks.

[27]  Yoshua Bengio,et al.  Semi-supervised Learning by Entropy Minimization , 2004, CAP.

[28]  O. Mangasarian,et al.  Semi-superyised support vector machines for unlabeled data classification , 2001 .

[29]  Alain Biem,et al.  Minimum classification error training for online handwriting recognition , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Matthias Seeger,et al.  Learning from Labeled and Unlabeled Data , 2010, Encyclopedia of Machine Learning.

[31]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[32]  Tommi S. Jaakkola,et al.  Partially labeled classification with Markov random walks , 2001, NIPS.

[33]  Alexander Zien,et al.  Semi-Supervised Classification by Low Density Separation , 2005, AISTATS.

[34]  Yuan Qi,et al.  Hyperparameter and Kernel Learning for Graph Based Semi-Supervised Classification , 2005, NIPS.

[35]  S. Sathiya Keerthi,et al.  Optimization Techniques for Semi-Supervised Support Vector Machines , 2008, J. Mach. Learn. Res..

[36]  Mohamed Cheriet,et al.  Model selection for the LS-SVM. Application to handwriting recognition , 2009, Pattern Recognit..

[37]  S. Sathiya Keerthi,et al.  Branch and Bound for Semi-Supervised Support Vector Machines , 2006, NIPS.

[38]  Jason Weston,et al.  Large Scale Transductive SVMs , 2006, J. Mach. Learn. Res..

[39]  Kok Seng Chua,et al.  Efficient computations for large least square support vector machine classifiers , 2003, Pattern Recognit. Lett..

[40]  Martial Hebert,et al.  Semi-Supervised Self-Training of Object Detection Models , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[41]  Tom Michael Mitchell,et al.  The Role of Unlabeled Data in Supervised Learning , 2004 .

[42]  Gavin C. Cawley,et al.  Fast exact leave-one-out cross-validation of sparse least-squares support vector machines , 2004, Neural Networks.

[43]  Gavin C. Cawley,et al.  Leave-One-Out Cross-Validation Based Model Selection Criteria for Weighted LS-SVMs , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[44]  Theodoros Damoulas,et al.  Multiclass Relevance Vector Machines: Sparsity and Accuracy , 2010, IEEE Transactions on Neural Networks.

[45]  Huanhuan Chen,et al.  Probabilistic Classification Vector Machines , 2009, IEEE Transactions on Neural Networks.

[46]  Mohamed Cheriet,et al.  Genetic algorithm–based training for semi-supervised SVM , 2010, Neural Computing and Applications.

[47]  Peter Tino,et al.  IEEE Transactions on Neural Networks , 2009 .

[48]  Johan A. K. Suykens,et al.  Benchmarking Least Squares Support Vector Machine Classifiers , 2004, Machine Learning.

[49]  Johan A. K. Suykens,et al.  Sparse least squares Support Vector Machine classifiers , 2000, ESANN.

[50]  Gavin C. Cawley,et al.  Improved sparse least-squares support vector machines , 2002, Neurocomputing.

[51]  Thorsten Joachims,et al.  Transductive Inference for Text Classification using Support Vector Machines , 1999, ICML.