Accelerating Convergence of Large-scale Optimization Algorithms

Several recent engineering applications in multi-agent systems, communication networks, and machine learning deal with decision problems that can be formulated as optimization problems. For many of ...

[1]  Reinhard Diestel,et al.  Graph Theory , 1997 .

[2]  Iman Shames,et al.  Formation control and coordination of autonomous agents , 2010 .

[3]  João M. F. Xavier,et al.  Distributed ADMM for model predictive control and congestion control , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[4]  Yurii Nesterov,et al.  Double Smoothing Technique for Large-Scale Linearly Constrained Convex Optimization , 2012, SIAM J. Optim..

[5]  A. Raghunathan,et al.  ADMM for Convex Quadratic Programs: Linear Convergence and Infeasibility Detection , 2014, 1411.7288.

[6]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[7]  Martin Hilbert,et al.  The World’s Technological Capacity to Store, Communicate, and Compute Information , 2011, Science.

[8]  Angelia Nedic,et al.  Distributed Optimization Over Time-Varying Directed Graphs , 2015, IEEE Trans. Autom. Control..

[9]  Brian D. O. Anderson,et al.  Analysis of accelerated gossip algorithms , 2009, CDC.

[10]  Nasser Yazdani,et al.  An analytical model of delay in multi-hop wireless ad hoc networks , 2011, Wirel. Networks.

[11]  Henrik Sandberg,et al.  Optimal scaling of the ADMM algorithm for distributed quadratic programming , 2013, 52nd IEEE Conference on Decision and Control.

[12]  Richard G. Baraniuk,et al.  Fast Alternating Direction Optimization Methods , 2014, SIAM J. Imaging Sci..

[13]  Emiliano Dall'Anese,et al.  Fast Consensus by the Alternating Direction Multipliers Method , 2011, IEEE Transactions on Signal Processing.

[14]  R. Varga,et al.  Chebyshev semi-iterative methods, successive overrelaxation iterative methods, and second order Richardson iterative methods , 1961 .

[15]  Björn Johansson,et al.  On Distributed Optimization in Networked Systems , 2008 .

[16]  Yurii Nesterov,et al.  Primal-dual subgradient methods for convex problems , 2005, Math. Program..

[17]  Euhanna Ghadimi,et al.  Hidden Terminal-Aware Contention Resolution with an Optimal Distribution , 2011, 2011 IEEE Eighth International Conference on Mobile Ad-Hoc and Sensor Systems.

[18]  Ambuj Tewari,et al.  Composite objective mirror descent , 2010, COLT 2010.

[19]  Jane J. Ye,et al.  Optimizing Condition Numbers , 2009, SIAM J. Optim..

[20]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[21]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2009, Found. Comput. Math..

[22]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[23]  Asuman E. Ozdaglar,et al.  Constrained Consensus and Optimization in Multi-Agent Networks , 2008, IEEE Transactions on Automatic Control.

[24]  Karl Henrik Johansson,et al.  Distributed MPC Via Dual Decomposition and Alternative Direction Method of Multipliers , 2012, ArXiv.

[25]  Euhanna Ghadimi,et al.  On the Optimal Step-size Selection for the Alternating Direction Method of Multipliers* , 2012 .

[26]  Pablo Soldati,et al.  On Cross-Layer Design and Resource Scheduling in Wireless Networks , 2009 .

[27]  Eric M. Yeatman,et al.  Urban Sensor Data Streams: London 2013 , 2013, IEEE Internet Computing.

[28]  Stephen P. Boyd,et al.  Metric Selection in Douglas-Rachford Splitting and ADMM , 2014 .

[29]  Junfeng Yang,et al.  Alternating Direction Algorithms for 1-Problems in Compressive Sensing , 2009, SIAM J. Sci. Comput..

[30]  Benjamin Recht,et al.  Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints , 2014, SIAM J. Optim..

[31]  Leon S. Lasdon,et al.  Optimization Theory of Large Systems , 1970 .

[32]  Dimitri P. Bertsekas,et al.  Convex Analysis and Optimization , 2003 .

[33]  C. D. Meyer,et al.  Convergent Powers of a Matrix With Applications to Iterative Methods for Singular Linear Systems , 1975 .

[34]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.

[35]  Thomas Brox,et al.  iPiasco: Inertial Proximal Algorithm for Strongly Convex Optimization , 2015, Journal of Mathematical Imaging and Vision.

[36]  Stephen P. Boyd,et al.  Fast linear iterations for distributed averaging , 2003, 42nd IEEE International Conference on Decision and Control (IEEE Cat. No.03CH37475).

[37]  Bart De Schutter,et al.  Accelerated gradient methods and dual decomposition in distributed model predictive control , 2013, Autom..

[38]  Mathew D. Penrose,et al.  Random Geometric Graphs , 2003 .

[39]  Jose Araujo,et al.  Design, Implementation and Validation of Resource-Aware and Resilient Wireless Networked Control Systems , 2014 .

[40]  Cristobal Guzman,et al.  On lower complexity bounds for large-scale smooth convex optimization , 2013, J. Complex..

[41]  Fan Chung,et al.  Spectral Graph Theory , 1996 .

[42]  Asuman E. Ozdaglar,et al.  On the O(1=k) convergence of asynchronous distributed alternating Direction Method of Multipliers , 2013, 2013 IEEE Global Conference on Signal and Information Processing.

[43]  Stephen P. Boyd,et al.  An ADMM Algorithm for a Class of Total Variation Regularized Estimation Problems , 2012, 1203.1828.

[44]  Zeyuan Allen-Zhu,et al.  Linear Coupling of Gradient and Mirror Descent: A Novel, Simple Interpretation of Nesterov's Accelerated Method , 2014 .

[45]  Steven H. Low,et al.  Optimization flow control—I: basic algorithm and convergence , 1999, TNET.

[46]  Ion Necoara,et al.  Rate Analysis of Inexact Dual First-Order Methods Application to Dual Decomposition , 2014, IEEE Transactions on Automatic Control.

[47]  Frank Kelly,et al.  Rate control for communication networks: shadow prices, proportional fairness and stability , 1998, J. Oper. Res. Soc..

[48]  Yurii Nesterov,et al.  Gradient methods for minimizing composite functions , 2012, Mathematical Programming.

[49]  David M. Young,et al.  Second-degree iterative methods for the solution of large linear systems , 1972 .

[50]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[51]  Lin Xiao,et al.  An Accelerated Randomized Proximal Coordinate Gradient Method and its Application to Regularized Empirical Risk Minimization , 2015, SIAM J. Optim..

[52]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[53]  Stephen P. Boyd,et al.  Notes on Decomposition Methods , 2008 .

[54]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[55]  J. Hiriart-Urruty,et al.  Convex analysis and minimization algorithms , 1993 .

[56]  Euhanna Ghadimi,et al.  A metric for opportunistic routing in duty cycled wireless sensor networks , 2012, 2012 9th Annual IEEE Communications Society Conference on Sensor, Mesh and Ad Hoc Communications and Networks (SECON).

[57]  S. Butler.  Eigenvalues and structures of graphs , 2008 .

[58]  E.J. Candes,et al.  An Introduction To Compressive Sampling , 2008, IEEE Signal Processing Magazine.

[59]  Hamed Farhadi,et al.  Coordinated Transmission for Wireless Interference Networks , 2014 .

[60]  Qing Ling,et al.  On the Linear Convergence of the ADMM in Decentralized Consensus Optimization , 2013, IEEE Transactions on Signal Processing.

[61]  Stephen J. Wright,et al.  Duality-based algorithms for total-variation-regularized image restoration , 2010, Comput. Optim. Appl..

[62]  Zhaosong Lu,et al.  An Accelerated Proximal Coordinate Gradient Method and its Application to Regularized Empirical Risk Minimization , 2014, 1407.1296.

[63]  James T. Kwok,et al.  Asynchronous Distributed ADMM for Consensus Optimization , 2014, ICML.

[64]  David J. Goodman,et al.  Power control for wireless data , 1999, 1999 IEEE International Workshop on Mobile Multimedia Communications (MoMuC'99) (Cat. No.99EX384).

[65]  John N. Tsitsiklis,et al.  Distributed Asynchronous Deterministic and Stochastic Gradient Optimization Algorithms , 1984, 1984 American Control Conference.

[66]  Karl Meerbergen,et al.  The Quadratic Eigenvalue Problem , 2001, SIAM Rev..

[67]  M. Johansson,et al.  Multi-Step Gradient Methods for Networked Optimization , 2013, IEEE Transactions on Signal Processing.

[68]  M. Johansson,et al.  Accelerated gradient methods for networked optimization , 2011 .

[69]  Stephen P. Boyd,et al.  Optimal Scaling of a Gradient Method for Distributed Resource Allocation , 2006 .

[70]  Manfred Morari,et al.  Computational aspects of distributed optimization in model predictive control , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[71]  M. H. Wright.  The interior-point revolution in optimization: History, recent developments, and lasting consequences , 2004 .

[72]  Stephen P. Boyd,et al.  Simultaneous routing and resource allocation via dual decomposition , 2004, IEEE Transactions on Communications.

[73]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[74]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[75]  João Pedro Hespanha,et al.  A Survey of Recent Results in Networked Control Systems , 2007, Proceedings of the IEEE.

[76]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[77]  Panganamala Ramana Kumar,et al.  RHEINISCH-WESTFÄLISCHE TECHNISCHE HOCHSCHULE AACHEN , 2001 .

[78]  Karl Henrik Johansson,et al.  The quadruple-tank process: a multivariable laboratory process with an adjustable zero , 2000, IEEE Trans. Control. Syst. Technol..

[79]  Daniel Boley,et al.  Local Linear Convergence of the Alternating Direction Method of Multipliers on Quadratic or Linear Programs , 2013, SIAM J. Optim..

[80]  Asuman Ozdaglar,et al.  Cooperative distributed multi-agent optimization , 2010, Convex Optimization in Signal Processing and Communications.

[81]  Hamid Reza Feyzmahdavian,et al.  Contractive interference functions and rates of convergence of distributed power control laws , 2012, ICC.

[82]  Hongbin Wang,et al.  Scaled Heavy-Ball Acceleration of the Richardson-Lucy Algorithm for 3D Microscopy Image Restoration , 2014, IEEE Transactions on Image Processing.

[83]  Stephen P. Boyd,et al.  QoS and fairness constrained convex optimization of resource allocation for wireless cellular and ad hoc networks , 2002, Proceedings.Twenty-First Annual Joint Conference of the IEEE Computer and Communications Societies.

[84]  Marc Teboulle,et al.  Performance of first-order methods for smooth convex minimization: a novel approach , 2012, Mathematical Programming.

[85]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[86]  Bo Wahlberg,et al.  An ADMM algorithm for solving ℓ1 regularized MPC , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[87]  Philip Wolfe,et al.  An algorithm for quadratic programming , 1956 .

[88]  A. Nemirovski,et al.  Interior-point methods for optimization , 2008, Acta Numerica.

[89]  Stephen P. Boyd,et al.  SegmentationofARX-modelsUsingSum-of-Norms Regularization ? , 2010 .

[90]  Yurii Nesterov,et al.  First-order methods of smooth convex optimization with inexact oracle , 2013, Mathematical Programming.

[91]  S. K. Zavriev,et al.  Heavy-ball method in nonconvex optimization problems , 1993 .

[92]  Pangun Park,et al.  Modeling, Analysis and Design of Wireless Sensor Network Protocols , 2011 .

[93]  Xuemin Shen,et al.  A Dual-Decomposition-Based Resource Allocation for OFDMA Networks With Imperfect CSI , 2010, IEEE Transactions on Vehicular Technology.

[94]  Pascal Bianchi,et al.  Asynchronous distributed optimization using a randomized alternating direction method of multipliers , 2013, 52nd IEEE Conference on Decision and Control.

[95]  Carlo Fischione,et al.  Fast-Lipschitz optimization with wireless sensor networks applications , 2011, Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks.

[96]  Reza Olfati-Saber,et al.  Consensus and Cooperation in Networked Multi-Agent Systems , 2007, Proceedings of the IEEE.

[97]  Ting Kei Pong,et al.  Minimizing Condition Number via Convex Programming , 2011, SIAM J. Matrix Anal. Appl..

[98]  Jonathan Eckstein.  Parallel alternating direction multiplier decomposition of convex programs , 1994 .

[99]  Andre Teixeira,et al.  Toward Cyber-Secure and Resilient Networked Control Systems , 2014 .

[100]  Assad Alam,et al.  Fuel-Efficient Heavy-Duty Vehicle Platooning , 2014 .

[101]  Georgios B. Giannakis,et al.  Consensus-Based Distributed Support Vector Machines , 2010, J. Mach. Learn. Res..

[102]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[103]  Dimitri P. Bertsekas,et al.  Incremental proximal methods for large scale convex optimization , 2011, Math. Program..

[104]  A. Robert Calderbank,et al.  Layering as Optimization Decomposition: A Mathematical Theory of Network Architectures , 2007, Proceedings of the IEEE.

[105]  Y. Nesterov.  A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[106]  Michael G. Rabbat,et al.  Optimization and Analysis of Distributed Averaging With Short Node Memory , 2009, IEEE Transactions on Signal Processing.

[107]  Piet Van Mieghem,et al.  Graph Spectra for Complex Networks , 2010 .

[108]  Mohammad Reza Gholami.  Wireless Sensor Network Positioning Techniques , 2013 .

[109]  Matti Latva-aho,et al.  Distributed SINR balancing for MISO downlink systems via the alternating direction method of multipliers , 2013, 2013 11th International Symposium and Workshops on Modeling and Optimization in Mobile, Ad Hoc and Wireless Networks (WiOpt).

[110]  Christian A. Larsson.  Application-oriented experiment design for industrial model predictive control , 2014 .

[111]  Yu-Chi Ho,et al.  A Class of Center-Free Resource Allocation Algorithms 1 , 1980 .

[112]  Andrea Simonetto,et al.  Distributed Estimation and Control for Robotic Networks , 2012 .

[113]  Piergiuseppe Di Marco.  Modeling and Design of Wireless Protocols for Networked Control Applications , 2012 .

[114]  Lin Xiao,et al.  Dual Averaging Methods for Regularized Stochastic Learning and Online Optimization , 2009, J. Mach. Learn. Res..

[115]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[116]  Euhanna Ghadimi,et al.  Opportunistic Routing in Low Duty-Cycle Wireless Sensor Networks , 2014, ACM Trans. Sens. Networks.

[117]  B. V. Dean,et al.  Studies in Linear and Non-Linear Programming. , 1959 .

[118]  Daniel A. Spielman,et al.  Accelerated Gossip Algorithms for Distributed Computation , 2006 .

[119]  Xiangrui Meng,et al.  Accelerating Nesterov's Method for Strongly Convex Functions with Lipschitz Gradient , 2011, 1109.6058.

[120]  Jonathan Eckstein.  Augmented Lagrangian and Alternating Direction Methods for Convex Optimization: A Tutorial and Some Illustrative Computational Results , 2012 .

[121]  Asuman E. Ozdaglar,et al.  Distributed Subgradient Methods for Multi-Agent Optimization , 2009, IEEE Transactions on Automatic Control.

[122]  E. I. Jury,et al.  Theory and application of the z-transform method , 1965 .

[123]  Zhenhua Zou,et al.  Real-time communication in wireless lossy networks , 2014 .

[124]  Liam Murphy,et al.  Parallel and distributed state estimation , 1995 .

[125]  Boris Polyak.  Some methods of speeding up the convergence of iteration methods , 1964 .

[126]  Pablo Soldati,et al.  Modular Design of Jointly Optimal Controllers and Forwarding Policies for Wireless Control , 2012, IEEE Transactions on Automatic Control.

[127]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[128]  Euhanna Ghadimi,et al.  The ADMM algorithm for distributed averaging: Convergence rates and optimal parameter selection , 2014, 2014 48th Asilomar Conference on Signals, Systems and Computers.

[129]  Stephen P. Boyd,et al.  A Splitting Method for Optimal Control , 2013, IEEE Transactions on Control Systems Technology.

[130]  Euhanna Ghadimi,et al.  Low power, low delay: Opportunistic routing meets duty cycling , 2012, 2012 ACM/IEEE 11th International Conference on Information Processing in Sensor Networks (IPSN).

[131]  Jie Lin,et al.  Coordination of groups of mobile autonomous agents using nearest neighbor rules , 2003, IEEE Trans. Autom. Control..

[132]  José M. Bioucas-Dias,et al.  Restoration of Poissonian Images Using Alternating Direction Optimization , 2010, IEEE Transactions on Image Processing.

[133]  Geoffrey Ye Li,et al.  A survey of energy-efficient wireless communications , 2013, IEEE Communications Surveys & Tutorials.