Distributed majorization-minimization for Laplacian regularized problems

We consider the problem of minimizing a block separable convex function ( possibly nondifferentiable, and including constraints ) plus Laplacian regularization, a problem that arises in applications including model fitting, regularizing stratified models, and multi-period portfolio optimization. We develop a distributed majorization-minimization method for this general problem, and derive a complete, self-contained, general, and simple proof of convergence. Our method is able to scale to very large problems, and we illustrate our approach on two applications, demonstrating its scalability and accuracy.

[1]  Alan L. Yuille,et al.  The Concave-Convex Procedure , 2003, Neural Computation.

[2]  Junbin Gao,et al.  Laplacian Regularized Low-Rank Representation and Its Applications , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  D. Hunter,et al.  A Tutorial on MM Algorithms , 2004 .

[4]  Stephen P. Boyd,et al.  CVXPY: A Python-Embedded Modeling Language for Convex Optimization , 2016, J. Mach. Learn. Res..

[5]  Pascal Frossard,et al.  Learning Laplacian Matrix in Smooth Graph Signal Representations , 2014, IEEE Transactions on Signal Processing.

[6]  Stephen P. Boyd,et al.  Fast Model Predictive Control Using Online Optimization , 2010, IEEE Transactions on Control Systems Technology.

[7]  Jeffrey A. Fessler,et al.  Accelerating Ordered Subsets Image Reconstruction for X-ray CT Using Spatially Nonuniform Optimization Transfer , 2013, IEEE Transactions on Medical Imaging.

[8]  Dejan Slepcev,et al.  Analysis of $p$-Laplacian Regularization in Semi-Supervised Learning , 2017, SIAM J. Math. Anal..

[9]  Jeffrey A. Fessler,et al.  Fast Parallel MR Image Reconstruction via B1-Based, Adaptive Restart, Iterative Soft Thresholding Algorithms (BARISTA) , 2015, IEEE Transactions on Medical Imaging.

[10]  Chris Godsil,et al.  The Laplacian of a Graph , 2001 .

[11]  J. Borwein,et al.  Convex Analysis And Nonlinear Optimization , 2000 .

[12]  Kenneth Lange,et al.  MM optimization algorithms , 2016 .

[13]  Stephen P. Boyd.  Plenary talk: Performance bounds and suboptimal policies for multi-period investment , 2013, MED.

[14]  Prabhu Babu,et al.  Majorization-Minimization Algorithms in Signal Processing, Communications, and Machine Learning , 2017, IEEE Transactions on Signal Processing.

[15]  Stephen P. Boyd,et al.  Multi-Period Portfolio Optimization with Constraints and Transaction Costs , 2009 .

[16]  Mikhail Belkin,et al.  Laplacian Support Vector Machines Trained in the Primal , 2009, J. Mach. Learn. Res..

[17]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[18]  Stephen P. Boyd,et al.  Network Inference via the Time-Varying Graphical Lasso , 2017, KDD.

[19]  Rok Sosic,et al.  SnapVX: A Network-Based Convex Optimization Solver , 2017, J. Mach. Learn. Res..

[20]  Stephen P. Boyd,et al.  Multi-Period Trading via Convex Optimization , 2017, Found. Trends Optim..

[21]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[22]  H. Waelbroeck,et al.  Optimal Execution of Portfolio Transactions with Short‐Term Alpha , 2013 .

[23]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[24]  Ami Wiesel,et al.  Joint Estimation of Inverse Covariance Matrices Lying in an Unknown Subspace , 2017, IEEE Transactions on Signal Processing.

[25]  E. Levitan,et al.  A Maximum a Posteriori Probability Expectation Maximization Algorithm for Image Reconstruction in Emission Tomography , 1987, IEEE Transactions on Medical Imaging.

[26]  K. Lange,et al.  The MM Alternative to EM , 2010, 1104.2203.

[27]  Antonio Ortega,et al.  Graph Learning From Data Under Laplacian and Structural Constraints , 2016, IEEE Journal of Selected Topics in Signal Processing.

[28]  R. Tibshirani,et al.  Covariance‐regularized regression and classification for high dimensional problems , 2009, Journal of the Royal Statistical Society. Series B, Statistical methodology.

[29]  M. Rothschild,et al.  Arbitrage, Factor Structure, and Mean-Variance Analysis on Large Asset Markets , 1983 .

[30]  Gene Cheung,et al.  Graph Laplacian Regularization for Image Denoising: Analysis in the Continuous Domain , 2016, IEEE Transactions on Image Processing.

[31]  Kenneth Lange,et al.  Sharp quadratic majorization in one dimension , 2009, Comput. Stat. Data Anal..

[32]  Alexandre d'Aspremont,et al.  Model Selection Through Sparse Max Likelihood Estimation Model Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data , 2022 .

[33]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[34]  F. Clarke.  Optimization And Nonsmooth Analysis , 1983 .

[35]  Stephen J. Wright,et al.  Warm-Start Strategies in Interior-Point Methods for Linear Programming , 2002, SIAM J. Optim..

[36]  K. Brown,et al.  Graduate Texts in Mathematics , 1982 .

[37]  R. Tyrrell Rockafellar,et al.  Convex Analysis , 1970, Princeton Landmarks in Mathematics and Physics.

[38]  Jeffrey A. Fessler,et al.  An Expanded Theoretical Treatment of Iteration-Dependent Majorize-Minimize Algorithms , 2007, IEEE Transactions on Image Processing.

[39]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[40]  Zhi-Quan Luo,et al.  A Unified Convergence Analysis of Block Successive Minimization Methods for Nonsmooth Optimization , 2012, SIAM J. Optim..

[41]  Patrick Danaher,et al.  The joint graphical lasso for inverse covariance estimation across multiple classes , 2011, Journal of the Royal Statistical Society. Series B, Statistical methodology.

[42]  Tong Zhang,et al.  Learning on Graph with Laplacian Regularization , 2006, NIPS.

[43]  B. Schölkopf,et al.  Graph Laplacian Regularization for Large-Scale Semidefinite Programming , 2007 .

[44]  G. M.,et al.  Partial Differential Equations I , 2023, Applied Mathematical Sciences.

[45]  Philip A. Chou,et al.  Graph Signal Processing – A Probabilistic Framework , 2016 .

[46]  Stephen P. Boyd,et al.  Conic Optimization via Operator Splitting and Homogeneous Self-Dual Embedding , 2013, Journal of Optimization Theory and Applications.

[47]  R. Tibshirani,et al.  Sparse inverse covariance estimation with the graphical lasso. , 2008, Biostatistics.

[48]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[49]  Gabriele Steidl,et al.  First order algorithms in variational image processing , 2014, ArXiv.

[50]  Stephen P. Boyd,et al.  OSQP: an operator splitting solver for quadratic programs , 2017, 2018 UKACC 12th International Conference on Control (CONTROL).