Non-smoothness in classification problems

We review the role played by non-smooth optimization techniques in many recent applications in classification area. Starting from the classical concept of linear separability in binary classification, we recall the more general concepts of polyhedral, ellipsoidal and max–min separability. Finally we focus our attention on the support vector machine (SVM) approach and on the more recent transductive SVM technique.

[1]  Jason Weston,et al.  Trading convexity for scalability , 2006, ICML.

[2]  Robert Mifflin,et al.  A -algorithm for convex minimization , 2005, Math. Program..

[3]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[4]  Annabella Astorino,et al.  Ellipsoidal separation for classification problems , 2005, Optim. Methods Softw..

[5]  Antonio Fuduli,et al.  Tuning Strategy for the Proximity Parameter in Convex Minimization , 2006 .

[6]  Rafail N. Gasimov,et al.  Separation via polyhedral conic functions , 2006, Optim. Methods Softw..

[7]  Claude Lemaréchal,et al.  Variable metric bundle methods: From conceptual to implementable forms , 1997, Math. Program..

[8]  Giorgio Gallo,et al.  A Bundle Type Dual-Ascent Approach to Linear Multicommodity Min-Cost Flow Problems , 1999, INFORMS J. Comput..

[10]  Johannes O. Royset,et al.  Algorithms with Adaptive Smoothing for Finite Minimax Problems , 2003 .

[11]  J. Frédéric Bonnans,et al.  A family of variable metric proximal methods , 1995, Math. Program..

[12]  Antonio Fuduli,et al.  Analysis of regularization techniques in convex nondifferentiable optimization , 1997 .

[13]  Krzysztof C. Kiwiel,et al.  Proximity control in bundle methods for convex nondifferentiable minimization , 1990, Math. Program..

[14]  Thorsten Joachims,et al.  Transductive Inference for Text Classification using Support Vector Machines , 1999, ICML.

[15]  Jochem Zowe,et al.  A Version of the Bundle Idea for Minimizing a Nonsmooth Function: Conceptual Idea, Convergence Analysis, Numerical Results , 1992, SIAM J. Optim..

[16]  Claude Lemaréchal,et al.  Practical Aspects of the Moreau-Yosida Regularization: Theoretical Preliminaries , 1997, SIAM J. Optim..

[17]  Paola Cappanera,et al.  Symmetric and Asymmetric Parallelization of a Cost-Decomposition Algorithm for Multicommodity Flow Problems , 2003, INFORMS J. Comput..

[18]  K. G. Murty,et al.  Nonsmooth optimization , 2007 .

[19]  Ayhan Demiriz,et al.  Semi-Supervised Support Vector Machines , 1998, NIPS.

[20]  Krzysztof C. Kiwiel,et al.  A tilted cutting plane proximal bundle method for convex nondifferentiable optimization , 1991, Oper. Res. Lett..

[21]  K. Kiwiel.  Efficiency of Proximal Bundle Methods , 2000 .

[22]  D. Medhi.  Parallel bundle-based decomposition for large-scale structured mathematical programming problems , 1990 .

[23]  Manlio Gaudioso,et al.  Quadratic approximations in convex nondifferentiable optimization , 1991 .

[24]  Yurii Nesterov,et al.  New variants of bundle methods , 1995, Math. Program..

[25]  Antonio Fuduli,et al.  Fixed and virtual stability center methods for convex nonsmooth minimization , 2000 .

[26]  Fabio Tardella.  Piecewise concavity and discrete approaches to continuous minimax problems , 2000 .

[27]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[28]  J. Hiriart-Urruty,et al.  Convex analysis and minimization algorithms , 1993 .

[29]  Adil M. Bagirov.  Minimization Methods for One Class of Nonsmooth Functions and Calculation of Semi-Equilibrium Prices , 1999 .

[30]  M. F. Monaco,et al.  Variants to the cutting plane approach for convex nondifferentiable optimization , 1992 .

[31]  Adil M. Bagirov,et al.  New algorithms for multi-class cancer diagnosis using tumor gene expression signatures , 2003, Bioinform..

[32]  C. Lemaréchal,et al.  THE U -LAGRANGIAN OF A CONVEX FUNCTION , 1996 .

[33]  C. Lemaréchal.  An extension of davidon methods to non differentiable problems , 1975 .

[34]  P. Wolfe,et al.  A METHOD OF CONJUGATE SUBGRADIENTS FOR , 1975 .

[35]  Carlo Vercellis,et al.  Accurately learning from few examples with a polyhedral classifier , 2007, Comput. Optim. Appl..

[36]  Vladimir F. Demyanov,et al.  Mathematical diagnostics via nonsmooth analysis , 2005, Optim. Methods Softw..

[37]  Per Olov Lindberg,et al.  A descent proximal level bundle method for convex nondifferentiable optimization , 1995, Oper. Res. Lett..

[38]  M. Fukushima,et al.  Globally Convergent BFGS Method for Nonsmooth Convex Optimization1 , 2000 .

[39]  Robert Mifflin,et al.  On VU-theory for Functions with Primal-Dual Gradient Structure , 2000, SIAM J. Optim..

[40]  Nimrod Megiddo,et al.  On the complexity of polyhedral separability , 1988, Discret. Comput. Geom..

[41]  O. Mangasarian,et al.  Semi-Supervised Support Vector Machines for Unlabeled Data Classification , 2001 .

[42]  K. Kiwiel.  Methods of Descent for Nondifferentiable Optimization , 1985 .

[43]  Adrian S. Lewis,et al.  A Robust Gradient Sampling Algorithm for Nonsmooth, Nonconvex Optimization , 2005, SIAM J. Optim..

[44]  Antonio Fuduli,et al.  A DC piecewise affine model and a bundling technique in nonconvex nonsmooth minimization , 2004, Optim. Methods Softw..

[45]  M. Gaudioso,et al.  Nonsmooth Problems in Mathematical Diagnostics , 2001 .

[46]  Peng Sun,et al.  Computation of Minimum Volume Covering Ellipsoids , 2002, Oper. Res..

[47]  O. Mangasarian.  Linear and Nonlinear Separation of Patterns by Linear Programming , 1965 .

[48]  Roberto Solis-Oba,et al.  Local Search , 2007, Handbook of Approximation Algorithms and Metaheuristics.

[49]  Defeng Sun,et al.  Quasi-Newton Bundle-Type Methods for Nondifferentiable Convex Optimization , 1998, SIAM J. Optim..

[50]  Robert Mifflin,et al.  A quasi-second-order proximal bundle algorithm , 1996, Math. Program..

[51]  Antonio Fuduli,et al.  Minimizing Nonconvex Nonsmooth Functions via Cutting Planes and Proximity Control , 2003, SIAM J. Optim..

[52]  Manlio Gaudioso,et al.  Nonsmooth optimization methods for parallel decomposition of multicommodity flow problems , 1993, Ann. Oper. Res..

[53]  G. Stavroulakis,et al.  Nonconvex Optimization in Mechanics: Algorithms, Heuristics and Engineering Applications , 1997 .

[54]  P. Neittaanmäki,et al.  Nonsmooth Optimization: Analysis and Algorithms with Applications to Optimal Control , 1992 .

[55]  Antonio Frangioni,et al.  Generalized Bundle Methods , 2002, SIAM J. Optim..

[56]  P. Wolfe.  Note on a method of conjugate subgradients for minimizing nondifferentiable functions , 1974 .

[57]  Marko Mäkelä,et al.  Survey of Bundle Methods for Nonsmooth Optimization , 2002, Optim. Methods Softw..

[58]  Philip Wolfe,et al.  Note on a method of conjugate subgradients for minimizing nondifferentiable functions , 1974, Math. Program..

[59]  R. Mifflin.  A modification and an extension of Lemarechal’s algorithm for nonsmooth minimization , 1982 .

[60]  D. Bertsekas.  Nondifferentiable optimization via approximation , 1975 .

[61]  Claude Lemaréchal,et al.  An Algorithm for Minimizing Convex Functions , 1974, IFIP Congress.

[62]  A. Bagirov,et al.  Discrete Gradient Method: Derivative-Free Method for Nonsmooth Optimization , 2008 .

[63]  J. B. Rosen.  Pattern separation by convex programming , 1965 .

[64]  Annabella Astorino,et al.  Polyhedral Separability Through Successive LP , 2002 .

[65]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[66]  A. Bagirov.  Continuous Subdifferential Approximations and Their Applications , 2003 .

[67]  Annabella Astorino,et al.  Nonsmooth Optimization Techniques for Semisupervised Classification , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[68]  Masao Fukushima,et al.  A Globally and Superlinearly Convergent Algorithm for Nonsmooth Convex Minimization , 1996, SIAM J. Optim..

[69]  Masao Fukushima,et al.  A descent algorithm for nonsmooth convex optimization , 1984, Math. Program..

[70]  Xiaojun Chen,et al.  Proximal quasi-Newton methods for nondifferentiable convex optimization , 1999, Math. Program..

[71]  Manlio Gaudioso,et al.  A bundle type approach to the unconstrained minimization of convex nonsmooth functions , 1982, Math. Program..

[72]  Robert Mifflin,et al.  𝒱𝒰-smoothness and proximal point results for some nonconvex functions , 2004, Optim. Methods Softw..

[73]  Michal Kočvara,et al.  Nonsmooth approach to optimization problems with equilibrium constraints : theory, applications, and numerical results , 1998 .

[74]  O. Mangasarian,et al.  Robust linear programming discrimination of two linearly inseparable sets , 1992 .

[75]  M. Gaudioso,et al.  An approach to classification based on separation of sets by means of several ellipsoids , 2005 .

[76]  Krzysztof C. Kiwiel.  A bundle Bregman proximal method for convex nondifferentiable minimization , 1999, Math. Program..

[77]  Adil M. Bagirov,et al.  Max–min separability , 2005, Optim. Methods Softw..

[78]  Kristin P. Bennett,et al.  The Interplay of Optimization and Machine Learning Research , 2006, J. Mach. Learn. Res..

[79]  Krzysztof C. Kiwiel,et al.  Proximal level bundle methods for convex nondifferentiable optimization, saddle-point problems and variational inequalities , 1995, Math. Program..

[80]  Alexander Zien,et al.  Semi-Supervised Classification by Low Density Separation , 2005, AISTATS.

[81]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2003, ICTAI.

[82]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[83]  A. Rubinov,et al.  Unsupervised and supervised data classification via nonsmooth and global optimization , 2003 .

[84]  Michael C. Ferris,et al.  Partitioning mathematical programs for parallel solution , 1998, Math. Program..

[85]  F. Clarke.  Optimization And Nonsmooth Analysis , 1983 .

[86]  Xiaojun Chen,et al.  A preconditioning proximal newton method for nondifferentiable convex optimization , 1997, Math. Program..

[87]  A. Bagirov,et al.  A Global Optimization Approach to Classification , 2002 .

[88]  M. Guignard.  Lagrangean relaxation , 2003 .

[89]  Adil M. Bagirov,et al.  A new nonsmooth optimization algorithm for minimum sum-of-squares clustering problems , 2006, Eur. J. Oper. Res..

[90]  C. Charalambous,et al.  Non-linear minimax optimization as a sequence of least pth optimization with finite values of p , 1976 .