Random Coordinate Descent Methods for $\ell_{0}$ Regularized Convex Optimization

In this paper, we study the minimization of ℓ0 regularized optimization problems, where the objective function is composed of a smooth convex function and the ℓ0 regularization. We analyze optimality conditions for this nonconvex problem which lead to the separation of local minima into two restricted classes that are nested and around the set of global minima. Based on these restricted classes of local minima, we devise two new random coordinate descent type methods for solving these problems. In particular, we analyze the convergence properties of an iterative hard thresholding based random coordinate descent algorithm for which we prove that any limit point is a local minimum from the first restricted class of local minimizers. Then, we analyze the convergence of a random proximal alternating minimization method and show that any limit point of this algorithm is a local minima from the second restricted class of local minimizers. We also provide numerical experiments which show the superior behavior of our methods in comparison with the usual iterative hard thresholding algorithm.

[1]  Ion Necoara,et al.  Efficient random coordinate descent algorithms for large-scale structured nonconvex optimization , 2013, Journal of Global Optimization.

[2]  A. Monticelli,et al.  Electric power system state estimation , 2000, Proceedings of the IEEE.

[3]  Constantino M. Lagoa,et al.  A Sparsification Approach to Set Membership Identification of Switched Affine Systems , 2012, IEEE Transactions on Automatic Control.

[4]  Georgios B. Giannakis,et al.  Distributed Robust Power System State Estimation , 2012, IEEE Transactions on Power Systems.

[5]  Yonina C. Eldar,et al.  Sparsity Constrained Nonlinear Optimization: Optimality Conditions and Algorithms , 2012, SIAM J. Optim..

[6]  Ion Necoara,et al.  Random Coordinate Descent Algorithms for Multi-Agent Convex Optimization Over Networks , 2013, IEEE Transactions on Automatic Control.

[7]  Yurii Nesterov,et al.  Generalized Power Method for Sparse Principal Component Analysis , 2008, J. Mach. Learn. Res..

[8]  Fu Lin,et al.  Design of Optimal Sparse Feedback Gains via the Alternating Direction Method of Multipliers , 2011, IEEE Transactions on Automatic Control.

[9]  Tyrone L. Vincent,et al.  Concentration of measure inequalities for compressive toeplitz matrices with applications to detection and system identification , 2010, 49th IEEE Conference on Decision and Control (CDC).

[10]  Ion Necoara,et al.  A random coordinate descent algorithm for optimization problems with composite objective function and linear coupled constraints , 2013, Comput. Optim. Appl..

[11]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[12]  Lang Tong,et al.  Malicious Data Attacks on the Smart Grid , 2011, IEEE Transactions on Smart Grid.

[13]  H. Robbins,et al.  A CONVERGENCE THEOREM FOR NON NEGATIVE ALMOST SUPERMARTINGALES AND SOME APPLICATIONS**Research supported by NIH Grant 5-R01-GM-16895-03 and ONR Grant N00014-67-A-0108-0018. , 1971 .

[14]  Liuqing Yang,et al.  Phasor state estimation from PMU measurements with bad data , 2011, 2011 4th IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP).

[15]  Daniel E. Quevedo,et al.  Sparse Packetized Predictive Control for Networked Control Over Erasure Channels , 2013, IEEE Transactions on Automatic Control.

[16]  H. Fang,et al.  Identification of Switched Linear Systems via Sparse Optimization , 2015 .

[17]  Y. Nesterov,et al.  A RANDOM COORDINATE DESCENT METHOD ON LARGE-SCALE OPTIMIZATION PROBLEMS WITH LINEAR CONSTRAINTS , 2013 .

[18]  Mike E. Davies,et al.  Iterative Hard Thresholding for Compressed Sensing , 2008, ArXiv.

[19]  Peter Richtárik,et al.  Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function , 2011, Mathematical Programming.

[20]  Stephen P. Boyd,et al.  Segmentation of ARX-models using sum-of-norms regularization , 2010, Autom..

[21]  Zhaosong Lu,et al.  Iterative hard thresholding methods for l0\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_0$$\end{document} regulari , 2012, Mathematical Programming.

[22]  T. Blumensath,et al.  Iterative Thresholding for Sparse Approximations , 2008 .

[23]  Bhiksha Raj,et al.  Greedy sparsity-constrained optimization , 2011, 2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR).

[24]  Tyrone L. Vincent,et al.  Compressive System Identification in the Linear Time-Invariant framework , 2011, IEEE Conference on Decision and Control and European Control Conference.

[25]  Yurii Nesterov,et al.  Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems , 2012, SIAM J. Optim..

[26]  Roland Tóth,et al.  AnSDPapproach for l 0-minimization : application toARX model segmentation , 2013 .

[27]  Mila Nikolova,et al.  Description of the Minimizers of Least Squares Regularized with 퓁0-norm. Uniqueness of the Global Minimizer , 2013, SIAM J. Imaging Sci..

[28]  Francisco J. Prieto,et al.  A Sequential Quadratic Programming Algorithm Using an Incomplete Solution of the Subproblem , 1995, SIAM J. Optim..

[29]  Daniel E. Quevedo,et al.  Packetized predictive control for rate-limited networks via sparse representation , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).