MADMM: A Generic Algorithm for Non-smooth Optimization on Manifolds

Numerous problems in computer vision, pattern recognition, and machine learning are formulated as optimization with manifold constraints. In this paper, we propose the Manifold Alternating Directions Method of Multipliers (MADMM), an extension of the classical ADMM scheme for manifold-constrained non-smooth optimization problems. To our knowledge, MADMM is the first generic non-smooth manifold optimization method. We showcase our method on several challenging problems in dimensionality reduction, non-rigid correspondence, multi-modal clustering, and multidimensional scaling.

[1]  M. Powell A method for nonlinear constraints in minimization problems , 1969 .

[2]  M. Hestenes Multiplier and gradient methods , 1969 .

[3]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[4]  J. Berge,et al.  Orthogonal procrustes rotation for two or more matrices , 1977 .

[5]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[6]  Ulrich Pinkall,et al.  Computing Discrete Minimal Surfaces and Their Conjugates , 1993, Exp. Math..

[7]  Alan Edelman,et al.  The Geometry of Algorithms with Orthogonality Constraints , 1998, SIAM J. Matrix Anal. Appl..

[8]  O. P. Ferreira,et al.  Subgradient Algorithm on Riemannian Manifolds , 1998 .

[9]  Mikhail Belkin,et al.  Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering , 2001, NIPS.

[10]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[11]  N. Higham Computing the nearest correlation matrix—a problem from finance , 2002 .

[12]  Hinrich Schütze,et al.  Introduction to information retrieval , 2008 .

[13]  Pierre-Antoine Absil,et al.  Joint Diagonalization on the Oblique Manifold for Independent Component Analysis , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[14]  Sanjoy Dasgupta,et al.  Robust Euclidean embedding , 2006, ICML.

[15]  Ron Bekkerman,et al.  Multi-modal Clustering for Multimedia Collections , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  José Mario Martínez,et al.  On Augmented Lagrangian Methods with General Lower-Level Constraints , 2007, SIAM J. Optim..

[17]  Pierre-Antoine Absil,et al.  Trust-Region Methods on Riemannian Manifolds , 2007, Found. Comput. Math..

[18]  Yu. S. Ledyaev,et al.  Nonsmooth analysis on smooth manifolds , 2007 .

[19]  Felipe Alvarez,et al.  A Unifying Local Convergence Result for Newton's Method in Riemannian Manifolds , 2008, Found. Comput. Math..

[20]  F. Bach,et al.  Low-rank optimization for semidefinite convex problems , 2008, 0807.4423.

[21]  Stephen P. Boyd,et al.  Graph Implementations for Nonsmooth Convex Programs , 2008, Recent Advances in Learning and Control.

[22]  Richard G. Baraniuk,et al.  1-Bit compressive sensing , 2008, 2008 42nd Annual Conference on Information Sciences and Systems.

[23]  Alexander M. Bronstein,et al.  Numerical Geometry of Non-Rigid Shapes , 2009, Monographs in Computer Science.

[24]  Levent Tunçel,et al.  Optimization algorithms on matrix manifolds , 2009, Math. Comput..

[25]  Sewoong Oh,et al.  A Gradient Descent Algorithm on the Grassman Manifold for Matrix Completion , 2009, ArXiv.

[26]  Daphna Weinshall,et al.  Online Learning in The Manifold of Low-Rank Matrices , 2010, NIPS.

[27]  Hédy Attouch,et al.  Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Lojasiewicz Inequality , 2008, Math. Oper. Res..

[28]  Francis R. Bach,et al.  Low-Rank Optimization on the Cone of Positive Semidefinite Matrices , 2008, SIAM J. Optim..

[29]  Vladimir G. Kim,et al.  Blended intrinsic maps , 2011, ACM Trans. Graph..

[30]  Pierre-Antoine Absil,et al.  RTRMC: A Riemannian trust-region method for low-rank matrix completion , 2011, NIPS.

[31]  Silvere Bonnabel,et al.  Linear Regression under Fixed-Rank Constraints: A Riemannian Approach , 2011, ICML.

[32]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[33]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[34]  Xiaojun Chen,et al.  Smoothing methods for nonsmooth, nonconvex minimization , 2012, Math. Program..

[35]  Amit Singer,et al.  Eigenvector Synchronization, Graph Rigidity and the Molecule Problem , 2011, Information and inference : a journal of the IMA.

[36]  Ira Kemelmacher-Shlizerman,et al.  Global Motion Estimation from Point Matches , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[37]  Amit Singer,et al.  Exact and Stable Recovery of Rotations for Robust Synchronization , 2012, ArXiv.

[38]  Davide Eynard,et al.  Multimodal diffusion geometry by joint diagonalization of Laplacians , 2012, ArXiv.

[39]  Yaron Lipman,et al.  Sensor network localization by eigenvector synchronization over the euclidean group , 2012, TOSN.

[40]  Derek Nowrouzezahrai,et al.  Learning hatching for pen-and-ink illustration of surfaces , 2012, TOGS.

[41]  Hao Shen,et al.  Blind Source Separation With Compressively Sensed Linear Mixtures , 2011, IEEE Signal Processing Letters.

[42]  Yu Wang,et al.  Fast Regularization of Matrix-Valued Images , 2012, Efficient Algorithms for Global Optimization Methods in Computer Vision.

[43]  Alexander M. Bronstein,et al.  Coupled quasi‐harmonic bases , 2012, Comput. Graph. Forum.

[44]  Guangdong Feng,et al.  A Tensor Based Method for Missing Traffic Data Completion , 2013 .

[45]  Pascal Frossard,et al.  Clustering on Multi-Layer Graphs via Subspace Analysis on Grassmann Manifolds , 2013, IEEE Transactions on Signal Processing.

[46]  S. Osher,et al.  Compressed modes for variational problems in mathematics and physics , 2013, Proceedings of the National Academy of Sciences.

[47]  P. Thomas Fletcher,et al.  Probabilistic Principal Geodesic Analysis , 2013, NIPS.

[48]  Wotao Yin,et al.  A feasible method for optimization with orthogonality constraints , 2013, Math. Program..

[49]  Marcus A. Magnor,et al.  Compressed Manifold Modes for Mesh Processing , 2014, Comput. Graph. Forum.

[50]  Bamdev Mishra,et al.  Manopt, a matlab toolbox for optimization on manifolds , 2013, J. Mach. Learn. Res..

[51]  Leonidas J. Guibas,et al.  Functional map networks for analyzing and exploring large shape collections , 2014, ACM Trans. Graph..

[52]  Steven Thomas Smith,et al.  Optimization Techniques on Riemannian Manifolds , 2014, ArXiv.

[53]  Pierre-Antoine Absil,et al.  A Riemannian subgradient algorithm for economic dispatch with valve-point effect , 2014, J. Comput. Appl. Math..

[54]  Ivor W. Tsang,et al.  Riemannian Pursuit for Big Matrix Recovery , 2014, ICML.

[55]  Rongjie Lai,et al.  A Splitting Method for Orthogonality Constrained Problems , 2014, J. Sci. Comput..

[56]  Davide Eynard,et al.  Multimodal Manifold Analysis by Simultaneous Diagonalization of Laplacians , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Xavier Bresson,et al.  Functional correspondence by matrix completion , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).