An Efficient Re-scaled Perceptron Algorithm for Conic Systems

The classical perceptron algorithm is an elementary algorithm for solving a homogeneous linear inequality system Ax > 0, with many important applications in learning theory (e.g., [11,8]). A natural condition measure associated with this algorithm is the Euclidean width τ of the cone of feasible solutions, and the iteration complexity of the perceptron algorithm is bounded by 1/τ2. Dunagan and Vempala [5] have developed a re-scaled version of the perceptron algorithm with an improved complexity of O(n ln(1/τ)) iterations (with high probability), which is theoretically efficient in τ, and in particular is polynomial-time in the bit-length model. We explore extensions of the concepts of these perceptron methods to the general homogeneous conic system Ax ∈ int K where K is a regular convex cone. We provide a conic extension of the re-scaled perceptron algorithm based on the notion of a deep-separation oracle of a cone, which essentially computes a certificate of strong separation. We give a general condition under which the re-scaled perceptron algorithm is theoretically efficient, i.e., polynomial-time; this includes the cases when K is the cross-product of half-spaces, second-order cones, and the positive semi-definite cone.

[1]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[2]  L. Khachiyan Polynomial algorithms in linear programming , 1980 .

[3]  Richard Zippel,et al.  Proving Polynomial-Time for Sphere-Constrained Quadratic Programming , 1990 .

[4]  James Renegar,et al.  Linear programming, complexity theory and elementary functional analysis , 1995, Math. Program..

[5]  Tom Bylander,et al.  Learning linear threshold functions in the presence of classification noise , 1994, COLT '94.

[6]  A. Berman Cones, matrices and mathematical programming , 1973 .

[7]  Yurii Nesterov,et al.  Interior-point polynomial algorithms in convex programming , 1994, Siam studies in applied mathematics.

[8]  Robert M. Freund,et al.  Some characterizations and properties of the “distance to ill-posedness” and the condition measure of a conic linear system , 1999, Math. Program..

[9]  Yinyu Ye,et al.  On affine scaling algorithms for nonconvex quadratic programming , 1992, Math. Program..

[10]  Narendra Karmarkar,et al.  A new polynomial-time algorithm for linear programming , 1984, Comb..

[11]  I. S. Gradshteyn,et al.  Table of Integrals, Series, and Products , 1976 .

[12]  Robert M. Freund,et al.  Condition-Based Complexity of Convex Optimization in Conic Linear Form via the Ellipsoid Algorithm , 1999, SIAM J. Optim..

[13]  Gábor Pataki,et al.  On the Closedness of the Linear Image of a Closed Convex Cone , 2007, Math. Oper. Res..

[14]  J. Renegar Some perturbation theory for linear programming , 1994, Math. Program..

[15]  Robert M. Freund,et al.  On the Second-Order Feasibility Cone: Primal-Dual Representation and Efficient Projection , 2008, SIAM J. Optim..

[16]  Santosh S. Vempala,et al.  A simple polynomial-time rescaling algorithm for solving linear programs , 2004, STOC '04.

[17]  丸山 徹 Convex Analysisの二,三の進展について , 1977 .

[18]  James Renegar,et al.  On the worst-case arithmetic complexity of approximating zeros of polynomials , 1987, J. Complex..

[19]  Santosh S. Vempala,et al.  Solving convex programs by random walks , 2004, JACM.

[20]  Russ Bubley,et al.  Randomized algorithms , 1995, CSUR.

[21]  J. G. Pierce,et al.  Geometric Algorithms and Combinatorial Optimization , 2016 .

[22]  Yoav Freund,et al.  Large Margin Classification Using the Perceptron Algorithm , 1998, COLT.

[23]  Alan M. Frieze,et al.  A Polynomial-Time Algorithm for Learning Noisy Linear Threshold Functions , 1996, Algorithmica.

[24]  Henry Wolkowicz,et al.  Handbook of Semidefinite Programming , 2000 .

[25]  D. E. Raeside Table of Integrals, Series, and Products, edited by I. S. Gradshteyn and I. M. Ryzhik , 1981 .

[26]  Zhi-Quan Luo,et al.  Approximation Algorithms for Quadratic Programming , 1998, J. Comb. Optim..

[27]  L. Lovász,et al.  Geometric Algorithms and Combinatorial Optimization , 1981 .

[28]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[29]  L. G. H. Cijan A polynomial algorithm in linear programming , 1979 .

[30]  Albert B Novikoff,et al.  ON CONVERGENCE PROOFS FOR PERCEPTRONS , 1963 .