Fast ADMM Algorithm for Distributed Optimization with Adaptive Penalty

We propose new methods to speed up convergence of the Alternating Direction Method of Multipliers (ADMM), a common optimization tool in the context of large scale and distributed learning. The proposed method accelerates the speed of convergence by automatically deciding the constraint penalty needed for parameter consensus in each iteration. In addition, we also propose an extension of the method that adaptively determines the maximum number of iterations to update the penalty. We show that this approach effectively leads to an adaptive, dynamic network topology underlying the distributed optimization. The utility of the new penalty update schemes is demonstrated on both synthetic and real data, including a computer vision application of distributed structure from motion.

[1]  Pietro Perona,et al.  Evaluation of Features Detectors and Descriptors Based on 3D Objects , 2005, ICCV.

[2]  s-taiji.  Dual Averaging and Proximal Gradient Descent for Online Alternating Direction Multiplier Method , 2013 .

[3]  Vladimir Pavlovic,et al.  Distributed Probabilistic Learning for Camera Networks with Missing Data , 2012, NIPS.

[4]  Patrick Pérez,et al.  Distributed Non-convex ADMM-based inference in large-scale random fields , 2014, BMVC.

[5]  Wen Gao,et al.  Robust Estimation of 3D Human Poses from a Single Image , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Y. Nesterov.  A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[7]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[8]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[9]  Roberto Tron RenVidal.  A Benchmark for the Comparison of 3-D Motion Segmentation Algorithms , 2007 .

[10]  Michael E. Tipping,et al.  Probabilistic Principal Component Analysis , 1999 .

[11]  Nenghai Yu,et al.  Non-negative low rank and sparse graph for semi-supervised learning , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Richard G. Baraniuk,et al.  Fast Alternating Direction Optimization Methods , 2014, SIAM J. Imaging Sci..

[13]  Zhixun Su,et al.  Fixed-rank representation for unsupervised visual learning , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  René Vidal,et al.  Distributed computer vision algorithms through distributed averaging , 2011, CVPR 2011.

[15]  Dong Liu,et al.  Recognizing Complex Events in Videos by Learning Key Static-Dynamic Evidences , 2014, ECCV.

[16]  Alexander G. Gray,et al.  Stochastic Alternating Direction Method of Multipliers , 2013, ICML.

[17]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[18]  Dong Xu,et al.  Learning by Associating Ambiguously Labeled Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[20]  B. He,et al.  Alternating Direction Method with Self-Adaptive Penalty Parameters for Monotone Variational Inequalities , 2000 .

[21]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[22]  Iasonas Kokkinos,et al.  Fast and Exact: ADMM-Based Discriminative Shape Segmentation with Loopy Part Models , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[24]  Patrick Pérez,et al.  Distributed Non-Convex ADMM-inference in Large-scale Random Fields , 2014 .

[25]  Allen Y. Yang,et al.  A Convex Optimization Framework for Active Learning , 2013, 2013 IEEE International Conference on Computer Vision.

[26]  Georgios B. Giannakis,et al.  Distributed Clustering Using Wireless Sensor Networks , 2011, IEEE Journal of Selected Topics in Signal Processing.