RandomBoost: Simplified Multiclass Boosting Through Randomization

We propose a novel boosting approach to multiclass classification problems, in which multiple classes are distinguished by a set of random projection matrices in essence. The approach uses random projections to alleviate the proliferation of binary classifiers typically required to perform multiclass classification. The result is a multiclass classifier with a single vector-valued parameter, irrespective of the number of classes involved. Two variants of this approach are proposed. The first method randomly projects the original data into new spaces, while the second method randomly projects the outputs of learned weak classifiers. These methods are not only conceptually simple but also effective and easy to implement. A series of experiments on synthetic, machine learning, and visual recognition data sets demonstrate that our proposed methods could be compared favorably with existing multiclass boosting algorithms in terms of both the convergence rate and classification accuracy.

[1]  J. Friedman.  Special Invited Paper-Additive logistic regression: A statistical view of boosting , 2000 .

[2]  Barbara Caputo,et al.  Safety in numbers: Learning categories from few examples with multi model knowledge transfer , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[3]  Cordelia Schmid,et al.  Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[4]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[5]  Hiroshi Mamitsuka,et al.  Boosted Network Classifiers for Local Feature Selection , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[6]  Yoram Singer,et al.  Boosting with structural sparsity , 2009, ICML '09.

[7]  Aníbal R. Figueiras-Vidal,et al.  Real AdaBoost With Gate Controlled Fusion , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Sebastian Nowozin,et al.  On feature combination for multiclass object classification , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[9]  Wei-Ying Ma,et al.  Learning similarity measure for natural image retrieval with relevance feedback , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[10]  Trevor Hastie,et al.  Multi-class AdaBoost ∗ , 2009 .

[11]  Nicolás García-Pedrajas,et al.  Constructing Ensembles of Classifiers by Means of Weighted Instance Selection , 2009, IEEE Transactions on Neural Networks.

[12]  Piotr Indyk,et al.  Similarity Search in High Dimensions via Hashing , 1999, VLDB.

[13]  Haibo He,et al.  SSC: A Classifier Combination Method Based on Signal Strength , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[14]  Yan Ke,et al.  PCA-SIFT: a more distinctive representation for local image descriptors , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[15]  Carla E. Brodley,et al.  Random Projection for High Dimensional Data Clustering: A Cluster Ensemble Approach , 2003, ICML.

[16]  Jian Zhang,et al.  Fast Pedestrian Detection Using a Cascade of Boosted Covariance Features , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[17]  Ling Li,et al.  Multiclass boosting with repartitioning , 2006, ICML.

[18]  Joseph K. Bradley,et al.  FilterBoost: Regression and Classification on Large Datasets , 2007, NIPS.

[19]  Piotr Indyk,et al.  Approximate nearest neighbors: towards removing the curse of dimensionality , 1998, STOC '98.

[20]  Wei-Ying Ma,et al.  Scalable music recommendation by search , 2007, ACM Multimedia.

[21]  Yoram Singer,et al.  An Efficient Boosting Algorithm for Combining Preferences by , 2013 .

[22]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[23]  M. Narasimha Murty,et al.  Speeding up AdaBoost Classifier with Random Projection , 2009, 2009 Seventh International Conference on Advances in Pattern Recognition.

[24]  Dmitriy Fradkin,et al.  Experiments with random projections for machine learning , 2003, KDD '03.

[25]  Thomas G. Dietterich,et al.  Solving Multiclass Learning Problems via Error-Correcting Output Codes , 1994, J. Artif. Intell. Res..

[26]  Anton van den Hengel,et al.  A Scalable Stagewise Approach to Large-Margin Multiclass Loss-Based Boosting , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[27]  Xin Yao,et al.  Sparse Approximation Through Boosting for Learning Large Scale Kernel Machines , 2010, IEEE Transactions on Neural Networks.

[28]  Yoram Singer,et al.  Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers , 2000, J. Mach. Learn. Res..

[29]  Yoram Singer,et al.  Improved Boosting Algorithms Using Confidence-rated Predictions , 1998, COLT' 98.

[30]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[31]  Nick Barnes,et al.  Fast and Robust Object Detection Using Asymmetric Totally Corrective Boosting , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[32]  Heikki Mannila,et al.  Random projection in dimensionality reduction: applications to image and text data , 2001, KDD '01.

[33]  Robert E. Schapire,et al.  Using output codes to boost multiclass learning problems , 1997, ICML.

[34]  Yuan Yuan,et al.  Incremental threshold learning for classifier selection , 2012, Neurocomputing.

[35]  Cynthia Rudin,et al.  Margin-based Ranking and an Equivalence between AdaBoost and RankBoost , 2009, J. Mach. Learn. Res..

[36]  Yan Ke,et al.  Efficient Near-duplicate Detection and Sub-image Retrieval , 2004 .

[37]  Jian Zhang,et al.  Face detection from few training examples , 2008, 2008 15th IEEE International Conference on Image Processing.

[38]  Santosh S. Vempala,et al.  An algorithmic theory of learning: Robust concepts and random projection , 1999, Machine Learning.

[39]  Venkatesan Guruswami,et al.  Multiclass learning, boosting, and error-correcting codes , 1999, COLT '99.

[40]  Trevor Darrell,et al.  Fast pose estimation with parameter-sensitive hashing , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[41]  Tat-Jen Cham,et al.  Fast training and selection of Haar features using statistics in boosting-based face detection , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[42]  Chunhua Shen,et al.  Rapid face recognition using hashing , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[43]  Luca Maria Gambardella,et al.  Better Digit Recognition with a Committee of Simple Neural Nets , 2011, 2011 International Conference on Document Analysis and Recognition.

[44]  Ryan M. Rifkin,et al.  In Defense of One-Vs-All Classification , 2004, J. Mach. Learn. Res..

[45]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[46]  Chunhua Shen,et al.  Boosting Through Optimization of Margin Distributions , 2009, IEEE Transactions on Neural Networks.

[47]  Ayhan Demiriz,et al.  Linear Programming Boosting via Column Generation , 2002, Machine Learning.

[48]  Laurens van der Maaten,et al.  A New Benchmark Dataset for Handwritten Character Recognition , 2009 .

[49]  Chunhua Shen,et al.  On the Dual Formulation of Boosting Algorithms , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  Chunhua Shen,et al.  A direct formulation for totally-corrective multi-class boosting , 2011, CVPR 2011.

[51]  Peter L. Bartlett,et al.  Boosting Algorithms as Gradient Descent , 1999, NIPS.