Optimal Parameter Selection for the Alternating Direction Method of Multipliers (ADMM): Quadratic Problems

The alternating direction method of multipliers (ADMM) has emerged as a powerful technique for large-scale structured optimization. Despite many recent results on the convergence properties of ADMM, a quantitative characterization of the impact of the algorithm parameters on the convergence times of the method is still lacking. In this paper we find the optimal algorithm parameters that minimize the convergence factor of the ADMM iterates in the context of ℓ2-regularized minimization and constrained quadratic programming. Numerical examples show that our parameter selection rules significantly outperform existing alternatives in the literature.

[1]  Leon S. Lasdon,et al.  Optimization Theory of Large Systems , 1970 .

[2]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[3]  C. Kelley.  Iterative Methods for Linear and Nonlinear Equations , 1987 .

[4]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[5]  Jonathan Eckstein.  Parallel alternating direction multiplier decomposition of convex programs , 1994 .

[6]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[7]  Karl Henrik Johansson,et al.  The quadruple-tank process: a multivariable laboratory process with an adjustable zero , 2000, IEEE Trans. Control. Syst. Technol..

[8]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[9]  Mohammad Shahidehpour,et al.  Parallel and Distributed State Estimation , 2003 .

[10]  Stephen P. Boyd,et al.  Fast linear iterations for distributed averaging , 2003, 42nd IEEE International Conference on Decision and Control (IEEE Cat. No.03CH37475).

[11]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[12]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[13]  José M. Bioucas-Dias,et al.  Restoration of Poissonian Images Using Alternating Direction Optimization , 2010, IEEE Transactions on Image Processing.

[14]  Georgios B. Giannakis,et al.  Consensus-based distributed linear support vector machines , 2010, IPSN '10.

[15]  Stephen P. Boyd,et al.  Segmentation of ARX-models using sum-of-norms regularization , 2010, Autom..

[16]  Georgios B. Giannakis,et al.  Consensus-Based Distributed Support Vector Machines , 2010, J. Mach. Learn. Res..

[17]  Asuman E. Ozdaglar,et al.  Constrained Consensus and Optimization in Multi-Agent Networks , 2008, IEEE Transactions on Automatic Control.

[18]  Junfeng Yang,et al.  Alternating Direction Algorithms for 1-Problems in Compressive Sensing , 2009, SIAM J. Sci. Comput..

[19]  P. Aguiar,et al.  A Proof of Convergence For the Alternating Direction Method of Multipliers Applied to Polyhedral-Constrained Functions , 2011, 1112.2295.

[20]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[21]  Emiliano Dall'Anese,et al.  Fast Consensus by the Alternating Direction Multipliers Method , 2011, IEEE Transactions on Signal Processing.

[22]  Matti Latva-aho,et al.  Distributed resource allocation for MISO downlink systems via the alternating direction method of multipliers , 2012, ACSCC.

[23]  Stephen P. Boyd,et al.  An ADMM Algorithm for a Class of Total Variation Regularized Estimation Problems , 2012, 1203.1828.

[24]  Jonathan Eckstein.  Augmented Lagrangian and Alternating Direction Methods for Convex Optimization: A Tutorial and Some Illustrative Computational Results , 2012 .

[25]  Matti Latva-aho,et al.  Distributed SINR balancing for MISO downlink systems via the alternating direction method of multipliers , 2013, 2013 11th International Symposium and Workshops on Modeling and Optimization in Mobile, Ad Hoc and Wireless Networks (WiOpt).

[26]  P. Giselsson.  Improving Fast Dual Ascent for MPC - Part II: The Embedded Case , 2013, 1312.3013.

[27]  Daniel Boley,et al.  Local Linear Convergence of the Alternating Direction Method of Multipliers on Quadratic or Linear Programs , 2013, SIAM J. Optim..

[28]  M. Johansson,et al.  Multi-Step Gradient Methods for Networked Optimization , 2011, IEEE Transactions on Signal Processing.

[29]  Stephen P. Boyd,et al.  A Splitting Method for Optimal Control , 2013, IEEE Transactions on Control Systems Technology.

[30]  Henrik Sandberg,et al.  Optimal scaling of the ADMM algorithm for distributed quadratic programming , 2013, 52nd IEEE Conference on Decision and Control.

[31]  Qing Ling,et al.  On the Linear Convergence of the ADMM in Decentralized Consensus Optimization , 2013, IEEE Transactions on Signal Processing.

[32]  Richard G. Baraniuk,et al.  Fast Alternating Direction Optimization Methods , 2014, SIAM J. Imaging Sci..

[33]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[34]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.