A fast iterative algorithm for support vector data description

Support vector data description (SVDD) is a well known model for pattern analysis when only positive examples are reliable. SVDD is usually trained by solving a quadratic programming problem, which is time consuming. This paper formulates the Lagrangian of a simply modified SVDD model as a differentiable convex function over the nonnegative orthant. The resulting minimization problem can be solved by a simple iterative algorithm. The proposed algorithm is easy to implement, without requiring any particular optimization toolbox. Theoretical and experimental analysis show that the algorithm converges r-linearly to the unique minimum point. Extensive experiments on pattern classification were conducted, and compared to the quadratic programming based SVDD (QP-SVDD), the proposed approach is much more computationally efficient (hundreds of times faster) and yields similar performance in terms of receiver operating characteristic curve. Furthermore, the proposed method and QP-SVDD extract almost the same set of support vectors.

[1]  Songfeng Zheng,et al.  Smoothly approximated support vector domain description , 2016, Pattern Recognit..

[2]  Xinman Zhang,et al.  Efficient iris segmentation method with support vector domain description , 2009 .

[3]  David R. Musicant,et al.  Lagrangian Support Vector Machines , 2001, J. Mach. Learn. Res..

[4]  Robert P. W. Duin,et al.  Uniform Object Generation for Optimizing One-class Classifiers , 2002, J. Mach. Learn. Res..

[5]  Sang-Woong Lee,et al.  Low resolution face recognition based on support vector data description , 2006, Pattern Recognit..

[6]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[7]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[8]  Robert P. W. Duin,et al.  Data domain description using support vectors , 1999, ESANN.

[9]  David R. Musicant,et al.  Active set support vector regression , 2004, IEEE Transactions on Neural Networks.

[10]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[11]  Lorenzo Bruzzone,et al.  A Support Vector Domain Description Approach to Supervised Classification of Remote Sensing Images , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[12]  Dolf Talman,et al.  A new pivoting algorithm for the linear complementarity problem allowing for an arbitrary starting point , 1994, Math. Program..

[13]  Federico Girosi,et al.  An improved training algorithm for support vector machines , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[14]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[15]  Robert P. W. Duin,et al.  Support Vector Data Description , 2004, Machine Learning.

[16]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[17]  S. Balasundaram,et al.  Lagrangian support vector regression via unconstrained convex minimization , 2014, Neural Networks.

[18]  Richard W. Cottle,et al.  Linear Complementarity Problem. , 1992 .

[19]  Songfeng Zheng.  A fast algorithm for training support vector regression via smoothed primal function minimization , 2015, Int. J. Mach. Learn. Cybern..

[20]  Grace Wahba,et al.  Generalized Approximate Cross Validation For Support Vector Machines, Or, Another Way To Look At Mar , 1999 .

[21]  Nello Cristianini,et al.  Advances in Kernel Methods - Support Vector Learning , 1999 .

[22]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[23]  Muhamed Aganagic,et al.  Newton's method for linear complementarity problems , 1984, Math. Program..

[24]  Xizhao Wang,et al.  Fast Fuzzy Multicategory SVM Based on Support Vector Domain Description , 2008, Int. J. Pattern Recognit. Artif. Intell..

[25]  Paul Armand,et al.  A Feasible BFGS Interior Point Algorithm for Solving Convex Minimization Problems , 2000, SIAM J. Optim..

[26]  Richard W. Cottle,et al.  Linear Complementarity Problem , 2009, Encyclopedia of Optimization.

[27]  David M. W. Powers,et al.  Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation , 2011, ArXiv.

[28]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[29]  Daewon Lee,et al.  Domain described support vector classifier for multi-classification problems , 2007, Pattern Recognit..

[30]  Chih-Jen Lin,et al.  Coordinate Descent Method for Large-scale L2-loss Linear Support Vector Machines , 2008, J. Mach. Learn. Res..

[31]  Carl Gold,et al.  Model selection for support vector machine classification , 2002, Neurocomputing.

[32]  Olvi L. Mangasarian,et al.  Nonlinear Programming , 1969 .

[33]  David S. Doermann,et al.  Support Vector Data Description for image categorization from Internet images , 2008, 2008 19th International Conference on Pattern Recognition.

[34]  Robert P. W. Duin,et al.  Support vector domain description , 1999, Pattern Recognit. Lett..

[35]  Federico Girosi,et al.  Training support vector machines: an application to face detection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[36]  Hava T. Siegelmann,et al.  Support Vector Clustering , 2002, J. Mach. Learn. Res..

[37]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[38]  Glenn Fung,et al.  Finite Newton method for Lagrangian support vector machine classification , 2003, Neurocomputing.

[39]  Han Liu,et al.  Blockwise coordinate descent procedures for the multi-task lasso, with applications to neural semantic basis discovery , 2009, ICML '09.

[40]  A. Ruszczynski,et al.  Nonlinear Optimization , 2006 .