First order constrained optimization algorithms with feasibility updates

We propose first order algorithms for convex optimization problems where the feasible set is described by a large number of convex inequalities that is to be explored by subgradient projections. The first algorithm is an adaptation of a subgradient algorithm, and has convergence rate $1/\sqrt{k}$. The second algorithm has convergence rate 1/k when (1) one has linear metric inequality in the feasible set, (2) the objective function is strongly convex, differentiable and has Lipschitz gradient, and (3) it is easy to optimize the objective function on the intersection of two halfspaces. This second algorithm generalizes Haugazeau's algorithm. The third algorithm adapts the second algorithm when condition (3) is dropped. We give examples to show that the second algorithm performs poorly when the objective function is not strongly convex, or when the linear metric inequality is absent.

[1]  Frank Deutsch,et al.  The rate of convergence for the cyclic projections algorithm III: Regularity of convex sets , 2008, J. Approx. Theory.

[2]  Alexander Y. Kruger.  Weak Stationarity: Eliminating the gap between necessary and sufficient conditions , 2004 .

[3]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[4]  Xu Shusheng,et al.  Estimation of the convergence rate of Dykstra’s cyclic projections algorithm in polyhedral case , 2000 .

[5]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[6]  Wei Chen,et al.  A fast algorithm for solving a linear feasibility problem with application to Intensity-Modulated Radiation Therapy. , 2008, Linear algebra and its applications.

[7]  M. Raydan,et al.  Alternating Projection Methods , 2011 .

[8]  Donald Goldfarb,et al.  A numerically stable dual method for solving strictly convex quadratic programs , 1983, Math. Program..

[9]  许树声.  ESTIMATION OF THE CONVERGENCE RATE OF DYKSTRA’S CYCLIC PROJECTIONS ALGORITHM IN POLYHEDRAL CASE , 2000 .

[10]  Karin Schwab,et al.  Best Approximation In Inner Product Spaces , 2016 .

[11]  Frank Deutsch,et al.  The rate of convergence for the cyclic projections algorithm I: Angles between convex sets , 2006, J. Approx. Theory.

[12]  R. Dykstra,et al.  A Method for Finding Projections onto the Intersection of Convex Sets in Hilbert Spaces , 1986 .

[13]  F. Deutsch.  Best approximation in inner product spaces , 2001 .

[14]  J. Borwein,et al.  Techniques of variational analysis , 2005 .

[15]  Frank Deutsch,et al.  The rate of convergence for the cyclic projections algorithm II: Norms of nonlinear operators , 2006, J. Approx. Theory.

[16]  C. H. Jeffrey Pang.  Set intersection problems: supporting hyperplanes and quadratic programming , 2015, Math. Program..

[17]  Marcos Raydan,et al.  Dykstra's Algorithm and Robust Stopping Criteria , 2009, Encyclopedia of Optimization.

[18]  R. Dykstra.  An Algorithm for Restricted Least Squares Regression , 1983 .

[19]  Dimitri P. Bertsekas,et al.  Incremental constraint projection methods for variational inequalities , 2014, Math. Program..

[20]  A. Juditsky,et al.  5 First-Order Methods for Nonsmooth Convex Large-Scale Optimization , I : General Purpose Methods , 2010 .

[21]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[22]  Angelia Nedic,et al.  Random algorithms for convex minimization problems , 2011, Math. Program..

[23]  Heinz H. Bauschke,et al.  On Projection Algorithms for Solving Convex Feasibility Problems , 1996, SIAM Rev..

[24]  Shih-Ping Han,et al.  A successive projection method , 1988, Math. Program..

[25]  Hein Hundal,et al.  The rate of convergence of dykstra's cyclic projections algorithm: The polyhedral case , 1994 .

[26]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[27]  Heinz H. Bauschke,et al.  Projection algorithms and monotone operators , 1996 .