Online Optimization in Dynamic Environments: A Regret Analysis for Sparse Problems

Time-varying systems are a challenge in many scientific and engineering areas. Usually, estimation of time-varying parameters or signals must be performed online, which calls for the development of responsive online algorithms. In this paper, we consider this problem in the context of the sparse optimization; specifically, we consider the Elastic-net model. Following the rationale in [1], we propose a novel online algorithm and we theoretically prove that it is successful in terms of dynamic regret. We then show an application to recursive identification of time-varying autoregressive models, in the case when the number of parameters to be estimated is unknown. Numerical results show the practical efficiency of the proposed method.

[1]  Alexandre M. Bayen,et al.  Online Homotopy Algorithm for a Generalization of the LASSO , 2013, IEEE Transactions on Automatic Control.

[2]  Alessandro Chiuso,et al.  Online identification of time-varying systems: A Bayesian approach , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[3]  Justin K. Romberg,et al.  Discrete and Continuous-Time Soft-Thresholding for Dynamic Signal Recovery , 2014, IEEE Transactions on Signal Processing.

[4]  Mario Sznaier,et al.  A Randomized Algorithm for Parsimonious Model Identification , 2018, IEEE Transactions on Automatic Control.

[5]  Enrico Magli,et al.  PISTA: Parallel Iterative Soft Thresholding algorithm for sparse image recovery , 2013, 2013 Picture Coding Symposium (PCS).

[6]  Holger Rauhut,et al.  A Mathematical Introduction to Compressive Sensing , 2013, Applied and Numerical Harmonic Analysis.

[7]  S. Fosson.  POLITECNICO DI TORINO Repository ISTITUZIONALE Distributed Support Detection of Jointly Sparse Signals / , 2022 .

[8]  Enrico Magli,et al.  Distributed Iterative Thresholding for $\ell _{0}/\ell _{1}$ -Regularized Linear Inverse Problems , 2015, IEEE Transactions on Information Theory.

[9]  Tyrone L. Vincent,et al.  Compressive System Identification in the Linear Time-Invariant framework , 2011, IEEE Conference on Decision and Control and European Control Conference.

[10]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[11]  M. Salman Asif,et al.  Dynamic Updating for ` 1 Minimization , 2009 .

[12]  Enrico Magli,et al.  Distributed support detection of jointly sparse signals , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[13]  Shai Shalev-Shwartz,et al.  Online Learning and Online Convex Optimization , 2012, Found. Trends Mach. Learn..

[14]  Ambuj Tewari,et al.  Composite objective mirror descent , 2010, COLT 2010.

[15]  Tyrone L. Vincent,et al.  Compressive System Identification of LTI and LTV ARX models , 2011, IEEE Conference on Decision and Control and European Control Conference.

[16]  Aryan Mokhtari,et al.  Optimization in Dynamic Environments : Improved Regret Rates for Strongly Convex Problems , 2016 .

[17]  Dave Zachariah,et al.  Dynamic Iterative Pursuit , 2012, IEEE Transactions on Signal Processing.

[18]  Massimo Fornasier,et al.  Numerical Methods for Sparse Recovery , 2010 .

[19]  Yang Li,et al.  Identification of Time-Varying Systems Using Multi-Wavelet Basis Functions , 2011, IEEE Transactions on Control Systems Technology.

[20]  Volkan Cevher,et al.  Dynamic sparse state estimation using ℓ1-ℓ1 minimization: Adaptive-rate measurement bounds, algorithms and applications , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[21]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[22]  Enrico Magli,et al.  Distributed Recovery of Jointly Sparse Signals Under Communication Constraints , 2016, IEEE Transactions on Signal Processing.

[23]  Lennart Ljung,et al.  System Identification Via Sparse Multiple Kernel-Based Regularization Using Sequential Convex Optimization Techniques , 2014, IEEE Transactions on Automatic Control.

[24]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[25]  Enrico Magli,et al.  Online convex optimization meets sparsity , 2017 .

[26]  Elad Hazan,et al.  Logarithmic regret algorithms for online convex optimization , 2006, Machine Learning.

[27]  Teodoro Alamo,et al.  Slide Window Bounded-Error Time-Varying Systems Identification , 2016, IEEE Transactions on Automatic Control.

[28]  Martin Zinkevich,et al.  Online Convex Programming and Generalized Infinitesimal Gradient Ascent , 2003, ICML.

[29]  Philip Schniter,et al.  Dynamic Compressive Sensing of Time-Varying Signals Via Approximate Message Passing , 2012, IEEE Transactions on Signal Processing.

[30]  Christopher J. Rozell,et al.  Dynamic Filtering of Time-Varying Sparse Signals via $\ell _1$ Minimization , 2015, IEEE Transactions on Signal Processing.

[31]  Shahin Shahrampour,et al.  Distributed Online Optimization in Dynamic Environments Using Mirror Descent , 2016, IEEE Transactions on Automatic Control.

[32]  S. Bittanti,et al.  Bounded error identification of time-varying parameters by RLS techniques , 1994, IEEE Trans. Autom. Control..

[33]  Mehran Mesbahi,et al.  Online Distributed Convex Optimization on Dynamic Networks , 2014, IEEE Transactions on Automatic Control.

[34]  Elad Hazan,et al.  Introduction to Online Convex Optimization , 2016, Found. Trends Optim..

[35]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[36]  V. Umanità,et al.  Elastic-Net Regularization: Iterative Algorithms and Asymptotic Behavior of Solutions , 2010 .

[37]  Eduardo F. Camacho,et al.  Bounded error identification of systems with time-varying parameters , 2004, 2004 43rd IEEE Conference on Decision and Control (CDC) (IEEE Cat. No.04CH37601).

[38]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.