Large-scale nonsmooth optimization : variable metric bundle method with limited memory

Haarala, Marjo Large-Scale Nonsmooth Optimization: Variable Metric Bundle Method with Limited Memory Jyväskylä: University of Jyväskylä, 2004, 107 pages (Jyväskylä Studies in Computing, ISSN 1456-5390) ISBN 951-39-1908-0 Finnish summary diss. Many practical optimization problems involve nonsmooth (that is, not necessarily differentiable) functions of hundreds or thousands of variables. In such problems, the direct application of smooth gradient-based methods may lead to a failure due to the nonsmooth nature of the problem. On the other hand, none of the current general nonsmooth optimization methods is efficient in large-scale settings. The motivation of this work is to develop efficient and reliable solvers for large-scale nonsmooth optimization problems. In this thesis, we introduce a new limited memory bundle method for nonsmooth large-scale optimization. The new method is a hybrid of the variable metric bundle method and the limited memory variable metric methods, where the former has been developed for smalland medium-scale nonsmooth optimization and the latter have been developed for large-scale smooth optimization. The new limited memory bundle method aims at filling the gap that exists in the field of nonsmooth optimization with large numbers of variables. Besides describing the new limited memory bundle method in detail, we prove its global convergence for locally Lipschitz continuous objective functions, which are not supposed to be differentiable or convex. In addition, we give some modifications to the basic method in order to improve the accuracy of the method without losing much in its efficiency. The efficiency and reliability of the new method and its modifications are demonstrated with numerical experiments. The problems included in our experiments contain both academic test problems and practical applications arising in the field of nonsmooth large-scale optimization.

[1]  J. E. Kelley,et al.  The Cutting-Plane Method for Solving Convex Programs , 1960 .

[2]  Roger Fletcher,et al.  A Rapidly Convergent Descent Method for Minimization , 1963, Comput. J..

[3]  R. Fletcher,et al.  A New Approach to Variable Metric Algorithms , 1970, Comput. J..

[4]  R. Tyrrell Rockafellar,et al.  Convex Analysis , 1970, Princeton Landmarks in Mathematics and Physics.

[5]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 2. The New Algorithm , 1970 .

[6]  D. Shanno.  Conditioning of Quasi-Newton Methods for Function Minimization , 1970 .

[7]  D. Goldfarb.  A family of variable-metric methods derived by variational means , 1970 .

[8]  C. Lemaréchal.  An extension of davidon methods to non differentiable problems , 1975 .

[9]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[10]  P. Toint.  On sparse and symmetric matrix updating subject to a linear equation , 1977 .

[11]  D. F. Shanno,et al.  Matrix conditioning and nonlinear optimization , 1978, Math. Program..

[12]  A. Conn,et al.  An Efficient Method to Solve the Minimax Problem Directly , 1978 .

[13]  C. Lemaréchal.  Nonsmooth Optimization and Descent Methods , 1978 .

[14]  J. Nocedal.  Updating Quasi-Newton Matrices With Limited Storage , 1980 .

[15]  P. Toint,et al.  Partitioned variable metric updates for large structured optimization problems , 1982 .

[16]  F. Clarke.  Optimization And Nonsmooth Analysis , 1983 .

[17]  Albert G. Buckley,et al.  QN-like variable storage conjugate gradients , 1983, Math. Program..

[18]  Ladislav Luksan,et al.  Dual method for solving a special problem of quadratic programming as a subproblem at linearly constrained nonlinear minimax approximation , 1984, Kybernetika.

[19]  K. Kiwiel.  Methods of Descent for Nondifferentiable Optimization , 1985 .

[20]  Nilama Gupta.  A higher than first order algorithm for non-smooth constrained optimization (nonlinear programming, penalty approach, lipschitz junction) , 1985 .

[21]  Naum Zuselevich Shor,et al.  Minimization Methods for Non-Differentiable Functions , 1985, Springer Series in Computational Mathematics.

[22]  J. Nocedal,et al.  Global Convergence of a Class of Quasi-newton Methods on Convex Problems, Siam Some Global Convergence Properties of a Variable Metric Algorithm for Minimization without Exact Line Searches, Nonlinear Programming, Edited , 1996 .

[23]  A. Auslender.  Numerical methods for nondifferentiable convex optimization , 1987 .

[24]  P. Toint,et al.  Testing a class of methods for solving minimization problems with simple bounds on the variables , 1988 .

[25]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[26]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[27]  K. Kiwiel.  An ellipsoid trust region bundle method for nonsmooth convex minimization , 1989 .

[28]  C. Lemaréchal.  Nondifferentiable optimization , 1989 .

[29]  Claude Lemaréchal,et al.  Some numerical experiments with variable-storage quasi-Newton algorithms , 1989, Math. Program..

[30]  Helga Schramm,et al.  Eine Kombination von Bundle- und Trust-region-Verfahren zur Lösung nichtdifferenzierbarer Optimierungsprobleme , 1989 .

[31]  Krzysztof C. Kiwiel,et al.  Proximity control in bundle methods for convex nondifferentiable minimization , 1990, Math. Program..

[32]  Krzysztof C. Kiwiel.  Approximations in Decomposition of Large-Scale Convex Programs via a Nondifferentiable Optimization Method , 1990 .

[33]  William C. Davidon,et al.  Variable Metric Method for Minimization , 1959, SIAM J. Optim..

[34]  P. Neittaanmäki,et al.  Nonsmooth Optimization: Analysis and Algorithms with Applications to Optimal Control , 1992 .

[35]  Jorge Nocedal,et al.  Theory of algorithms for unconstrained optimization , 1992, Acta Numerica.

[36]  Jochem Zowe,et al.  A Version of the Bundle Idea for Minimizing a Nonsmooth Function: Conceptual Idea, Convergence Analysis, Numerical Results , 1992, SIAM J. Optim..

[37]  M. F. Monaco,et al.  Variants to the cutting plane approach for convex nondifferentiable optimization , 1992 .

[38]  Michael L. Overton,et al.  Large-Scale Optimization of Eigenvalues , 1990, SIAM J. Optim..

[39]  J. Hiriart-Urruty,et al.  Convex analysis and minimization algorithms , 1993 .

[40]  Jorge Nocedal,et al.  Analysis of a self-scaling quasi-Newton method , 1993, Math. Program..

[41]  L. Luksan.  Computational experience with known variable metric updates , 1994 .

[42]  Jorge Nocedal,et al.  Representations of quasi-Newton matrices and their use in limited memory methods , 1994, Math. Program..

[43]  Yurii Nesterov,et al.  New variants of bundle methods , 1995, Math. Program..

[44]  Per Olov Lindberg,et al.  A descent proximal level bundle method for convex nondifferentiable optimization , 1995, Oper. Res. Lett..

[45]  J. Frédéric Bonnans,et al.  A family of variable metric proximal methods , 1995, Math. Program..

[46]  J. Nocedal,et al.  A Limited Memory Algorithm for Bound Constrained Optimization , 1995, SIAM J. Sci. Comput..

[47]  Jorge Nocedal,et al.  Large Scale Unconstrained Optimization , 1997 .

[48]  Jorge Nocedal,et al.  On the Implementation of an Algorithm for Large-Scale Equality Constrained Optimization , 1998, SIAM J. Optim..

[49]  Kaisa Miettinen,et al.  Optimal Control of Continuous Casting by Nondifferentiable Multiobjective Optimization , 1998, Comput. Optim. Appl..

[50]  Jan Vlcek,et al.  A bundle-Newton method for nonsmooth unconstrained minimization , 1998, Math. Program..

[51]  Tamara G. Kolda,et al.  BFGS with Update Skipping and Varying Memory , 1998, SIAM J. Optim..

[52]  L. Luksan,et al.  Globally Convergent Variable Metric Method for Convex Nonsmooth Unconstrained Minimization1 , 1999 .

[53]  M. M. MÄKELÄ,et al.  Comparing Nonsmooth Nonconvex Bundle Methods in Solving Hemivariational Inequalities , 1999, J. Glob. Optim..

[54]  L. Luksan,et al.  Variable metric methods for unconstrainted optimization and nonlinear least squares , 2000 .

[55]  Masao Fukushima,et al.  On the Global Convergence of the BFGS Method for Nonconvex Unconstrained Optimization Problems , 2000, SIAM J. Optim..

[56]  Tommi Kärkkäinen,et al.  Comparison of formulations and solution methods for image restoration problems , 2001 .

[57]  Andreas Grothey,et al.  Decomposition methods for nonlinear nonconvex optimization problems , 2001 .

[58]  L. Luksan,et al.  Globally Convergent Variable Metric Method for Nonconvex Nondifferentiable Unconstrained Minimization , 2001 .

[59]  Jorge J. Moré,et al.  Digital Object Identifier (DOI) 10.1007/s101070100263 , 2001 .

[60]  Marko Mäkelä,et al.  Survey of Bundle Methods for Nonsmooth Optimization , 2002, Optim. Methods Softw..

[61]  Marc Teboulle,et al.  Mirror descent and nonlinear projected subgradient methods for convex optimization , 2003, Oper. Res. Lett..

[62]  Antonio Fuduli,et al.  A DC piecewise affine model and a bundling technique in nonconvex nonsmooth minimization , 2004, Optim. Methods Softw..

[63]  Kaisa Miettinen,et al.  New limited memory bundle method for large-scale nonsmooth optimization , 2004, Optim. Methods Softw..

[64]  Antonio Fuduli,et al.  Minimizing Nonconvex Nonsmooth Functions via Cutting Planes and Proximity Control , 2003, SIAM J. Optim..