Exploiting Universum data in AdaBoost using gradient descent

Recently, Universum data that does not belong to any class of the training data, has been applied for training better classifiers. In this paper, we address a novel boosting algorithm called UAdaBoost that can improve the classification performance of AdaBoost with Universum data. UAdaBoost chooses a function by minimizing the loss for labeled data and Universum data. The cost function is minimized by a greedy, stagewise, functional gradient procedure. Each training stage of UAdaBoost is fast and efficient. The standard AdaBoost weights labeled samples during training iterations while UAdaBoost gives an explicit weighting scheme for Universum samples as well. In addition, this paper describes the practical conditions for the effectiveness of Universum learning. These conditions are based on the analysis of the distribution of ensemble predictions over training samples. Experiments on handwritten digits classification and gender classification problems are presented. As exhibited by our experimental results, the proposed method can obtain superior performances over the standard AdaBoost by selecting proper Universum data.

[1]  Yong Shi,et al.  Regularized multiple-criteria linear programming with universum and its application , 2012, Neural Computing and Applications.

[2]  Ivor W. Tsang,et al.  This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 1 Domain Adaptation from Multiple Sources: A Domain- , 2022 .

[3]  Jason Weston,et al.  Inference with the Universum , 2006, ICML.

[4]  Gang Qian,et al.  Recognizing body poses using multilinear analysis and semi-supervised learning , 2009, Pattern Recognit. Lett..

[5]  Changshui Zhang,et al.  Selecting Informative Universum Sample for Semi-Supervised Learning , 2009, IJCAI.

[6]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[7]  Ayhan Demiriz,et al.  Exploiting unlabeled data in ensemble methods , 2002, KDD.

[8]  Vladimir Cherkassky,et al.  Gender classification of human faces using inference through contradictions , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[9]  Fei Wang,et al.  Semi-Supervised Classification with Universum , 2008, SDM.

[10]  Peter L. Bartlett,et al.  Boosting Algorithms as Gradient Descent in Function Space , 2007 .

[11]  Vladimir Cherkassky,et al.  Cost-Sensitive Universum-SVM , 2012, 2012 11th International Conference on Machine Learning and Applications.

[12]  Y. Freund,et al.  Discussion of the Paper \additive Logistic Regression: a Statistical View of Boosting" By , 2000 .

[13]  Gunnar Rätsch,et al.  Efficient Margin Maximizing with Boosting , 2005, J. Mach. Learn. Res..

[14]  Yong Shi,et al.  Twin support vector machine with Universum data , 2012, Neural Networks.

[15]  Hui Xue,et al.  Universum linear discriminant analysis , 2012 .

[16]  Dan Zhang,et al.  Document clustering with universum , 2011, SIGIR.

[17]  Peter L. Bartlett,et al.  Boosting Algorithms as Gradient Descent , 1999, NIPS.

[18]  Fumin Shen,et al.  {\cal U}Boost: Boosting with the Universum , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Bernhard Schölkopf,et al.  An Analysis of Inference with the Universum , 2007, NIPS.

[20]  Daoqiang Zhang,et al.  Ensemble Universum SVM Learning for Multimodal Classification of Alzheimer's Disease , 2013, MLMI.

[21]  Chunhua Shen,et al.  On the Dual Formulation of Boosting Algorithms , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Christophe Ambroise,et al.  Semi-supervised MarginBoost , 2001, NIPS.

[23]  Vladimir Vapnik,et al.  Estimation of Dependences Based on Empirical Data: Empirical Inference Science (Information Science and Statistics) , 2006 .

[24]  Wuyang Dai,et al.  Practical Conditions for Effectiveness of the Universum Learning , 2011, IEEE Transactions on Neural Networks.

[25]  V. Vapnik.  Estimation of Dependences Based on Empirical Data , 2006 .

[26]  Gang Qian,et al.  View-Invariant Pose Recognition Using Multilinear Analysis and the Universum , 2008, ISVC.