SI-ADMM: A stochastic inexact ADMM framework for resolving structured stochastic convex programs

We consider the resolution of the structured stochastic convex program: min equation such that Ax+By = b. To exploit problem structure and allow for developing distributed schemes, we propose an inexact stochastic generalization in which the subproblems are solved inexactly via stochastic approximation schemes. Based on this framework, we prove the following: (i) when the inexactness sequence satisfies suitable summability properties, the proposed stochastic inexact ADMM (SI-ADMM) scheme produces a sequence that converges to the unique solution almost surely; (ii) if the inexactness is driven to zero at a polynomial (geometric) rate, the sequence converges to the unique solution in a mean-squared sense at a prescribed polynomial (geometric) rate.

[1]  Arindam Banerjee,et al.  Online Alternating Direction Method (longer version) , 2013, ArXiv.

[2]  Hesamoddin Ahmadi On the Analysis of Data-driven and Distributed Algorithms for Convex Optimization Problems , 2016 .

[3]  G. Sapiro,et al.  A collaborative framework for 3D alignment and classification of heterogeneous subvolumes in cryo-electron tomography. , 2013, Journal of structural biology.

[4]  Stephen P. Boyd,et al.  Metric Selection in Douglas-Rachford Splitting and ADMM , 2014 .

[5]  Alexander G. Gray,et al.  Stochastic Alternating Direction Method of Multipliers , 2013, ICML.

[6]  A. Nedić,et al.  Regularized Iterative Stochastic Approximation Methods for Variational Inequality Problems , 2011 .

[7]  Alexander Shapiro,et al.  Lectures on Stochastic Programming: Modeling and Theory , 2009 .

[8]  Yi Ma,et al.  The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices , 2010, Journal of structural biology.

[9]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[10]  Angelia Nedic,et al.  Regularized Iterative Stochastic Approximation Methods for Stochastic Variational Inequality Problems , 2013, IEEE Transactions on Automatic Control.

[11]  Alexander Shapiro,et al.  Stochastic Approximation approach to Stochastic Programming , 2013 .

[12]  Angelia Nedic,et al.  On stochastic gradient and subgradient methods with adaptive steplength sequences , 2011, Autom..

[13]  H. Robbins A Stochastic Approximation Method , 1951 .

[14]  Jonathan Eckstein Splitting methods for monotone operators with applications to parallel optimization , 1989 .

[15]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[16]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[17]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[18]  José M. Bioucas-Dias,et al.  Fast Image Recovery Using Variable Splitting and Constrained Optimization , 2009, IEEE Transactions on Image Processing.

[19]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[20]  Zhixun Su,et al.  Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation , 2011, NIPS.