On the Minimization Over Sparse Symmetric Sets: Projections, Optimality Conditions, and Algorithms

We consider the problem of minimizing a general continuously differentiable function over symmetric sets under sparsity constraints. These type of problems are generally hard to solve because the sparsity constraint induces a combinatorial constraint into the problem, rendering the feasible set to be nonconvex. We begin with a study of the properties of the orthogonal projection operator onto sparse symmetric sets. Based on this study, we derive efficient methods for computing sparse projections under various symmetry assumptions. We then introduce and study three types of optimality conditions: basic feasibility, L -stationarity, and coordinatewise optimality. A hierarchy between the optimality conditions is established by using the results derived on the orthogonal projection operator. Methods for generating points satisfying the various optimality conditions are presented, analyzed, and finally tested on specific applications.

[1]  Coralia Cartis,et al.  A New and Improved Quantitative Recovery Analysis for Iterative Hard Thresholding Algorithms in Compressed Sensing , 2013, IEEE Transactions on Information Theory.

[2]  Bastian Goldlücke,et al.  Variational Analysis , 2014, Computer Vision, A Reference Guide.

[3]  Marc Teboulle,et al.  Conditional Gradient Algorithmsfor Rank-One Matrix Approximations with a Sparsity Constraint , 2011, SIAM Rev..

[4]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[5]  Yonina C. Eldar,et al.  Sparsity Constrained Nonlinear Optimization: Optimality Conditions and Algorithms , 2012, SIAM J. Optim..

[6]  Bhiksha Raj,et al.  Greedy sparsity-constrained optimization , 2011, 2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR).

[7]  T. Blumensath,et al.  Iterative Thresholding for Sparse Approximations , 2008 .

[8]  K. Kiwiel.  On Linear-Time Algorithms for the Continuous Quadratic Knapsack Problem , 2007 .

[9]  Marc Teboulle,et al.  Proximal alternating linearized minimization for nonconvex and nonsmooth problems , 2013, Mathematical Programming.

[10]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[11]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[12]  Akiko Takeda,et al.  Simultaneous pursuit of out-of-sample performance and sparsity in index tracking portfolios , 2012, Comput. Manag. Sci..

[13]  Jorge J. Moré,et al.  Computing a Trust Region Step , 1983 .

[14]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[15]  Stephen J. Wright,et al.  Computational Methods for Sparse Solution of Linear Inverse Problems , 2010, Proceedings of the IEEE.

[16]  Volkan Cevher,et al.  Sparse projections onto the simplex , 2012, ICML.

[17]  Yong Zhang,et al.  Sparse Approximation via Penalty Decomposition Methods , 2012, SIAM J. Optim..

[18]  Deanna Needell,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, ArXiv.

[19]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[20]  Zhaosong Lu,et al.  Iterative hard thresholding methods for l0\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_0$$\end{document} regulari , 2012, Mathematical Programming.

[21]  Terence Tao,et al.  The Dantzig selector: Statistical estimation when P is much larger than n , 2005, math/0506081.

[22]  Yonina C. Eldar,et al.  Structured Compressed Sensing: From Theory to Applications , 2011, IEEE Transactions on Signal Processing.

[23]  Mike E. Davies,et al.  Normalized Iterative Hard Thresholding: Guaranteed Stability and Performance , 2010, IEEE Journal of Selected Topics in Signal Processing.

[24]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[25]  Naihua Xiu,et al.  Gradient Support Projection Algorithm for Affine Feasibility Problem with Sparsity and Nonnegativity , 2014, 1406.7178.

[26]  Benar Fux Svaiter,et al.  Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods , 2013, Math. Program..

[27]  R. Rockafellar.  The theory of subgradients and its applications to problems of optimization : convex and nonconvex functions , 1981 .

[28]  Yonina C. Eldar,et al.  Introduction to Compressed Sensing , 2022 .

[29]  Mike E. Davies,et al.  Iterative Hard Thresholding for Compressed Sensing , 2008, ArXiv.