Multicategory angle-based large-margin classification.

Large-margin classifiers are popular methods for classification. Among existing simultaneous multicategory large-margin classifiers, a common approach is to learn k different functions for a k-class problem with a sum-to-zero constraint. Such a formulation can be inefficient. We propose a new multicategory angle-based large-margin classification framework. The proposed angle-based classifiers consider a simplex-based prediction rule without the sum-to-zero constraint, and enjoy more efficient computation. Many binary large-margin classifiers can be naturally generalized for multicategory problems through the angle-based framework. Theoretical and numerical studies demonstrate the usefulness of the angle-based methods.

[1]  Trevor Hastie,et al.  Regularization Paths for Generalized Linear Models via Coordinate Descent. , 2010, Journal of statistical software.

[2]  M. Yuan,et al.  Reinforced Multicategory Support Vector Machines , 2011 .

[3]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[4]  Peter L. Bartlett,et al.  Classification with a Reject Option using a Hinge Loss , 2008, J. Mach. Learn. Res..

[5]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[6]  Nello Cristianini,et al.  An introduction to Support Vector Machines , 2000 .

[7]  Yichao Wu,et al.  Nonlinear vertex discriminant analysis with reproducing kernels , 2012, Stat. Anal. Data Min..

[8]  Yufeng Liu,et al.  Multicategory large-margin unified machines , 2013, J. Mach. Learn. Res..

[9]  Trevor Hastie,et al.  Multi-class AdaBoost ∗ , 2009 .

[10]  Hao Helen Zhang,et al.  Hard or Soft Classification? Large-Margin Unified Machines , 2011, Journal of the American Statistical Association.

[11]  W. Wong,et al.  On ψ-Learning , 2003 .

[12]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[13]  Yufeng Liu,et al.  Multicategory ψ-Learning , 2006 .

[14]  Arnaud Doucet,et al.  A Framework for Kernel-Based Multi-Category Classification , 2007, J. Artif. Intell. Res..

[15]  Grace Wahba,et al.  Soft and hard classification by reproducing kernel Hilbert space methods , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[16]  Koby Crammer,et al.  On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines , 2002, J. Mach. Learn. Res..

[17]  Xiwu Lin,et al.  Smoothing spline ANOVA models for large data sets with Bernoulli observations and the randomized GACV , 2000 .

[18]  Ji Zhu,et al.  Kernel Logistic Regression and the Import Vector Machine , 2001, NIPS.

[19]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1997, EuroCOLT.

[20]  Nuno Vasconcelos,et al.  Multiclass Boosting: Theory and Algorithms , 2011, NIPS.

[21]  Xiaotong Shen,et al.  On L1-Norm Multiclass Support Vector Machines , 2007 .

[22]  Lifeng Wang,et al.  Generalization error for multi-class margin classification , 2007, 0708.3556.

[23]  Hao Helen Zhang,et al.  Robust Model-Free Multiclass Probability Estimation , 2010, Journal of the American Statistical Association.

[24]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[25]  Ambuj Tewari,et al.  On the Consistency of Multiclass Classification Methods , 2007, J. Mach. Learn. Res..

[26]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[27]  Michael I. Jordan,et al.  Convexity, Classification, and Risk Bounds , 2006 .

[28]  Tong Zhang.  Statistical behavior and consistency of classification methods based on convex risk minimization , 2003 .

[29]  Yufeng Liu,et al.  Probability estimation for large-margin classifiers , 2008 .

[30]  N. Zhang,et al.  Scan Statistics With Weighted Observations , 2007 .

[31]  K. Lange,et al.  An MM Algorithm for Multicategory Vertex Discriminant Analysis , 2008 .

[32]  Tong Zhang,et al.  Statistical Analysis of Some Multi-Category Large Margin Classification Methods , 2004, J. Mach. Learn. Res..

[33]  G. Wahba,et al.  Multicategory Support Vector Machines , Theory , and Application to the Classification of Microarray Data and Satellite Radiance Data , 2004 .

[34]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.