Sharing features in multi-class boosting via group sparsity

We present a novel formulation of fully corrective boosting for multi-class classification problems with the awareness of sharing features. Our multi-class boosting is solved in a single optimization problem. In order to share features across different classes, we introduce the mixed-norm regularization, which promotes group sparsity, into boosting. We then derive the Lagrange dual problems which enable us to design fully corrective multi-class algorithms using the primal-dual optimization technique. We show that sharing features across classes can improve classification performance and efficiency. We empirically show that in many cases, the proposed multi-class boosting generalizes better than a range of competing multi-class boosting algorithms due to the capability of feature sharing. Experimental results on machine learning data, visual scene and object recognition demonstrate the efficiency and effectiveness of proposed algorithms and validate our theoretical findings.

[1]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[2]  Cordelia Schmid,et al.  Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[3]  Yoram Singer,et al.  Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers , 2000, J. Mach. Learn. Res..

[4]  O. Chapelle.  Multi-Class Feature Selection with Support Vector Machines , 2008 .

[5]  Yoav Freund,et al.  An Adaptive Version of the Boost by Majority Algorithm , 1999, COLT '99.

[6]  Yoram Singer,et al.  Improved Boosting Algorithms Using Confidence-rated Predictions , 1998, COLT' 98.

[7]  Venkatesan Guruswami,et al.  Multiclass learning, boosting, and error-correcting codes , 1999, COLT '99.

[8]  Chunhua Shen,et al.  A direct formulation for totally-corrective multi-class boosting , 2011, CVPR 2011.

[9]  Chih-Jen Lin,et al.  Probability Estimates for Multi-class Classification by Pairwise Coupling , 2003, J. Mach. Learn. Res..

[10]  Jian Zhang,et al.  Face detection from few training examples , 2008, 2008 15th IEEE International Conference on Image Processing.

[11]  Thomas G. Dietterich,et al.  Solving Multiclass Learning Problems via Error-Correcting Output Codes , 1994, J. Artif. Intell. Res..

[12]  Feng Liu,et al.  Multi-class AdaBoost learning of facial feature selection through Grid Computing , 2010, 2010 IEEE 9th International Conference on Cyberntic Intelligent Systems.

[13]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[14]  Johannes Fürnkranz,et al.  Round Robin Classification , 2002, J. Mach. Learn. Res..

[15]  Ayhan Demiriz,et al.  Linear Programming Boosting via Column Generation , 2002, Machine Learning.

[16]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[17]  Andrew Zisserman,et al.  Scene Classification Using a Hybrid Generative/Discriminative Approach , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Stan Z. Li,et al.  FloatBoost learning and statistical face detection , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Antonio Torralba,et al.  Sharing Visual Features for Multiclass and Multiview Object Detection , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Yang Wang,et al.  Finding shareable informative patterns and optimal coding matrix for multiclass boosting , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[21]  Yoram Singer,et al.  Improved Boosting Algorithms Using Confidence-rated Predictions , 1998, COLT' 98.

[22]  Jian Zhang,et al.  Fast Pedestrian Detection Using a Cascade of Boosted Covariance Features , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[23]  Ling Li,et al.  Multiclass boosting with repartitioning , 2006, ICML.

[24]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[25]  James M. Rehg,et al.  CENTRIST: A Visual Descriptor for Scene Categorization , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Yoram Singer,et al.  Boosting with structural sparsity , 2009, ICML '09.

[27]  Chunhua Shen,et al.  On the Dual Formulation of Boosting Algorithms , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Robert E. Schapire,et al.  Using output codes to boost multiclass learning problems , 1997, ICML.

[29]  Trevor Hastie,et al.  Multi-class AdaBoost ∗ , 2009 .

[30]  Koby Crammer,et al.  On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines , 2002, J. Mach. Learn. Res..

[31]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.