A Riemannian Gradient Sampling Algorithm for Nonsmooth Optimization on Manifolds

In this paper, an optimization method for nonsmooth locally Lipschitz functions on complete Riemannian manifolds is presented. The method is based on approximating the subdifferential of the cost function at every iteration by the convex hull of transported gradients from tangent spaces at randomly generated nearby points to the tangent space of the current iterate and can hence be seen as a generalization of the well known gradient sampling algorithm to a Riemannian setting. A convergence result is obtained under the assumption that the cost function is bounded below and continuously differentiable on an open set of full measure and that the employed vector transport and retraction satisfy certain conditions, which hold, for instance, for the exponential map and parallel transport. Then with probability one the algorithm produces iterates at which the cost function is differentiable, and each cluster point of the iterates is a Clarke stationary point. Modifications yielding only $\varepsilon$-stationary ...

[1]  Wen Huang,et al.  A Broyden Class of Quasi-Newton Methods for Riemannian Optimization , 2015, SIAM J. Optim..

[2]  Bart Vandereycken,et al.  Low-rank tensor completion by Riemannian optimization , 2014 .

[3]  W. Huang,et al.  Line Search Algorithms for Locally Lipschitz Functions on Riemannian Manifolds , 2018, SIAM J. Optim..

[4]  Adrian S. Lewis,et al.  A Robust Gradient Sampling Algorithm for Nonsmooth, Nonconvex Optimization , 2005, SIAM J. Optim..

[5]  Krzysztof C. Kiwiel,et al.  Convergence of the Gradient Sampling Algorithm for Nonsmooth Nonconvex Optimization , 2007, SIAM J. Optim..

[6]  Bamdev Mishra,et al.  Manopt, a matlab toolbox for optimization on manifolds , 2013, J. Mach. Learn. Res..

[7]  Xiao-Qing Jin,et al.  A Riemannian Newton Algorithm for Nonlinear Eigenvalue Problems , 2015, SIAM J. Matrix Anal. Appl..

[8]  John Wright,et al.  Finding a Sparse Vector in a Subspace: Linear Sparsity Using Alternating Directions , 2014, IEEE Transactions on Information Theory.

[9]  P. Neittaanmäki,et al.  Nonsmooth Optimization Theory , 1992 .

[10]  Pascal Frossard,et al.  Clustering on Multi-Layer Graphs via Subspace Analysis on Grassmann Manifolds , 2013, IEEE Transactions on Signal Processing.

[11]  Berkant Savas,et al.  Quasi-Newton Methods on Grassmannians and Multilinear Approximations of Tensors , 2009, SIAM J. Sci. Comput..

[12]  Philipp Grohs,et al.  Total variation regularization on Riemannian manifolds by iteratively reweighted minimization , 2016 .

[13]  Robert E. Mahony,et al.  Optimization Algorithms on Matrix Manifolds , 2007 .

[14]  R. Adler,et al.  Newton's method on Riemannian manifolds and a geometric model for the human spine , 2002 .

[15]  R. Riddell,et al.  Minimax problems on Grassmann manifolds. Sums of eigenvalues , 1984 .

[16]  Pierre-Antoine Absil,et al.  Accelerated Line-search and Trust-region Methods , 2009, SIAM J. Numer. Anal..

[17]  Wen Huang,et al.  A Riemannian BFGS Method for Nonconvex Optimization Problems , 2015, ENUMATH.

[18]  L. Lathauwer,et al.  On the best low multilinear rank approximation of higher-order tensors , 2010 .

[19]  Sabine Van Huffel,et al.  Best Low Multilinear Rank Approximation of Higher-Order Tensors, Based on the Riemannian Trust-Region Scheme , 2011, SIAM J. Matrix Anal. Appl..

[20]  M. R. Pouryayevali,et al.  Generalized gradients and characterization of epi-Lipschitz sets in Riemannian manifolds , 2011 .

[21]  Pierre-Antoine Absil,et al.  A Riemannian subgradient algorithm for economic dispatch with valve-point effect , 2014, J. Comput. Appl. Math..

[22]  Wen Huang,et al.  Optimization algorithms on Riemannian manifolds with applications , 2013 .

[23]  F. Clarke.  Optimization And Nonsmooth Analysis , 1983 .

[24]  Gabriele Steidl,et al.  A Second Order Nonsmooth Variational Model for Restoring Manifold-Valued Images , 2015, SIAM J. Sci. Comput..

[25]  Philipp Grohs,et al.  ε-subgradient algorithms for locally lipschitz functions on Riemannian manifolds , 2015, Advances in Computational Mathematics.

[26]  Adrian S. Lewis,et al.  Nonsmooth optimization via quasi-Newton methods , 2012, Mathematical Programming.

[27]  Hiroyuki Sato,et al.  A Riemannian Optimization Approach to the Matrix Singular Value Decomposition , 2013, SIAM J. Optim..

[28]  P. Grohs,et al.  Nonsmooth trust region algorithms for locally Lipschitz functions on Riemannian manifolds , 2016 .

[29]  Bart Vandereycken,et al.  Low-Rank Matrix Completion by Riemannian Optimization , 2013, SIAM J. Optim..

[30]  Bamdev Mishra,et al.  Fixed-rank matrix factorizations and Riemannian low-rank optimization , 2012, Comput. Stat..

[31]  Alan Edelman,et al.  The Geometry of Algorithms with Orthogonality Constraints , 1998, SIAM J. Matrix Anal. Appl..

[32]  Felix J. Herrmann,et al.  Optimization on the Hierarchical Tucker manifold – Applications to tensor completion , 2014, Linear Algebra and its Applications.

[33]  U. Helmke,et al.  Nonsmooth Riemannian Optimization with Applications to Sphere Packing and Grasping , 2007 .

[34]  Stefan Vandewalle,et al.  A Riemannian Optimization Approach for Computing Low-Rank Solutions of Lyapunov Equations , 2010, SIAM J. Matrix Anal. Appl..

[35]  Michael M. Bronstein,et al.  MADMM: A Generic Algorithm for Non-smooth Optimization on Manifolds , 2015, ECCV.

[36]  L. Evans.  Measure theory and fine properties of functions , 1992 .

[37]  Bamdev Mishra,et al.  Low-Rank Optimization with Trace Norm Penalty , 2011, SIAM J. Optim..