Convergence Analysis of Alternating Projection Method for Nonconvex Sets

Alternating projection method has been used in a wide range of engineering applications since it is a gradient-free method (without requiring tuning the step size) and usually has fast speed of convergence. In this paper, we formalize two properties of proper, lower semi-continuous and semi-algebraic sets: the three-point property for all possible iterates and the local contraction property that serves as the non-expensiveness property of the projector but only for the iterates that are close enough to each other. Then by exploiting the geometric properties of the objective function around its critical point, i.e. the Kurdyka-Ł{ojasiewicz} (KL) property, we establish a new convergence analysis framework to show that if one set satisfies the three-point property and the other one obeys the local contraction property, the iterates generated by alternating projection method is a convergent sequence and converges to a critical point. We complete this study by providing convergence rate which depends on the explicit expression of the KL exponent. As a byproduct, we use our new analysis framework to recover the linear convergence rate of alternating projection method onto closed convex sets. To illustrate the power of our new framework, we provide new convergence result for a class of concrete applications: alternating projection method for designing structured tight frames that are widely used in sparse representation, compressed sensing and communication.

[1]  K. Kurdyka.  On gradients of functions definable in o-minimal structures , 1998 .

[2]  Heinz H. Bauschke,et al.  On Projection Algorithms for Solving Convex Feasibility Problems , 1996, SIAM Rev..

[3]  H. H. Rachford,et al.  On the numerical solution of heat conduction problems in two and three space variables , 1956 .

[4]  Robert W. Heath,et al.  Interference alignment via alternating minimization , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[5]  Aude Rondepierre,et al.  On Local Convergence of the Method of Alternating Projections , 2013, Foundations of Computational Mathematics.

[6]  Patrick L. Combettes,et al.  Convex set theoretic image recovery by extrapolated iterations of parallel subgradient projections , 1997, IEEE Trans. Image Process..

[7]  Amir Beck,et al.  On the Convergence of Alternating Minimization for Convex Programming with Applications to Iteratively Reweighted Least Squares and Decomposition Schemes , 2015, SIAM J. Optim..

[8]  Junfeng Yang,et al.  A New Alternating Minimization Algorithm for Total Variation Image Reconstruction , 2008, SIAM J. Imaging Sci..

[9]  Robert R. Meyer,et al.  Sufficient Conditions for the Convergence of Monotonic Mathematical Programming Algorithms , 1976, J. Comput. Syst. Sci..

[10]  Adrian S. Lewis,et al.  Local Linear Convergence for Alternating and Averaged Nonconvex Projections , 2009, Found. Comput. Math..

[11]  Luigi Grippo,et al.  On the convergence of the block nonlinear Gauss-Seidel method under convex constraints , 2000, Oper. Res. Lett..

[12]  D. Youla,et al.  Image Restoration by the Method of Convex Projections: Part 1ߞTheory , 1982, IEEE Transactions on Medical Imaging.

[13]  T. Chan,et al.  Convergence of the alternating minimization algorithm for blind deconvolution , 2000 .

[14]  Prateek Jain,et al.  Low-rank matrix completion using alternating minimization , 2012, STOC '13.

[15]  Mark D. Plumbley,et al.  Learning Incoherent Dictionaries for Sparse Approximation Using Iterative Projections and Rotations , 2013, IEEE Transactions on Signal Processing.

[16]  Thomas Strohmer,et al.  GRASSMANNIAN FRAMES WITH APPLICATIONS TO CODING AND COMMUNICATION , 2003, math/0301135.

[17]  Adrian S. Lewis,et al.  THEINEQUALITY FOR NONSMOOTH SUBANALYTIC FUNCTIONS WITH APPLICATIONS TO , 2007 .

[18]  Bruce W. Suter,et al.  From error bounds to the complexity of first-order descent methods for convex functions , 2015, Math. Program..

[19]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[20]  Heinz H. Bauschke,et al.  On cluster points of alternating projections , 2013, 1307.2712.

[21]  Heinz H. Bauschke,et al.  On the convergence of von Neumann's alternating projection algorithm for two sets , 1993 .

[22]  Amir Beck,et al.  On the Convergence of Block Coordinate Descent Type Methods , 2013, SIAM J. Optim..

[23]  Alfredo N. Iusem,et al.  Convergence results for an accelerated nonlinear cimmino algorithm , 1986 .

[24]  Michael Elad,et al.  Optimized Projections for Compressed Sensing , 2007, IEEE Transactions on Signal Processing.

[25]  Benar Fux Svaiter,et al.  Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods , 2013, Math. Program..

[26]  Robert E. Mahony,et al.  Convergence of the Iterates of Descent Methods for Analytic Cost Functions , 2005, SIAM J. Optim..

[27]  D. Russell Luke,et al.  Nonconvex Notions of Regularity and Convergence of Fundamental Algorithms for Feasibility Problems , 2012, SIAM J. Optim..

[28]  Urs Niesen,et al.  Adaptive Alternating Minimization Algorithms , 2007, ISIT.

[29]  Nélida E. Echebest,et al.  Incomplete oblique projections for solving large inconsistent linear systems , 2007, Math. Program..

[30]  Anthony Man-Cho So,et al.  Quadratic optimization with orthogonality constraint: explicit Łojasiewicz exponent and linear convergence of retraction-based line-search and stochastic variance-reduced gradient methods , 2018, Mathematical Programming.

[31]  Heinz H. Bauschke,et al.  On the local convergence of the Douglas–Rachford algorithm , 2014, 1401.6188.

[32]  Xiao Li,et al.  Designing robust sensing matrix for image compression , 2015, IEEE Transactions on Image Processing.

[33]  Zhihui Zhu,et al.  On Projection Matrix Optimization for Compressive Sensing Systems , 2013, IEEE Transactions on Signal Processing.

[34]  K. Tanabe.  Projection method for solving a singular system of linear equations and its applications , 1971 .

[35]  Dmitriy Drusvyatskiy,et al.  Transversality and Alternating Projections for Nonconvex Sets , 2014, Found. Comput. Math..

[36]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[37]  Adrian S. Lewis,et al.  Clarke Subgradients of Stratifiable Functions , 2006, SIAM J. Optim..

[38]  Anthony Man-Cho So,et al.  Quadratic Optimization with Orthogonality Constraints: Explicit Lojasiewicz Exponent and Linear Convergence of Line-Search Methods , 2015, ICML.

[39]  øöö Blockinø.  Phase retrieval, error reduction algorithm, and Fienup variants: A view from convex optimization , 2002 .

[40]  Hédy Attouch,et al.  On the convergence of the proximal algorithm for nonsmooth functions involving analytic features , 2008, Math. Program..

[41]  Stephen P. Boyd,et al.  Alternating Projections , .

[42]  Robert W. Heath,et al.  Designing structured tight frames via an alternating projection method , 2005, IEEE Transactions on Information Theory.

[43]  C. Byrne,et al.  A unified treatment of some iterative algorithms in signal processing and image reconstruction , 2003 .

[44]  Z.-Q. Luo,et al.  Error bounds and convergence analysis of feasible descent methods: a general approach , 1993, Ann. Oper. Res..

[45]  Mike E. Davies,et al.  Parametric Dictionary Design for Sparse Coding , 2009, IEEE Transactions on Signal Processing.

[46]  Hédy Attouch,et al.  Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Lojasiewicz Inequality , 2008, Math. Oper. Res..

[47]  M. J. D. Powell,et al.  On search directions for minimization algorithms , 1973, Math. Program..

[48]  Aggelos K. Katsaggelos,et al.  Construction of Incoherent Unit Norm Tight Frames With Application to Compressed Sensing , 2014, IEEE Transactions on Information Theory.

[49]  Matthew K. Tam.  Alternating Projection Methods Failure in the Absence of Convexity , 2012 .

[50]  P. Wedin.  Perturbation bounds in connection with singular value decomposition , 1972 .

[51]  Marc Teboulle,et al.  Proximal alternating linearized minimization for nonconvex and nonsmooth problems , 2013, Mathematical Programming.