Limited Memory Space Dilation and Reduction Algorithms

In this paper, we present variants of Shor and Zhurbenko's r-algorithm, motivated by the memoryless and limited memory updates for differentiable quasi-Newton methods. This well known r-algorithm, which employs a space dilation strategy in the direction of the difference between two successive subgradients, is recognized as being one of the most effective procedures for solving nondifferentiable optimization problems. However, the method needs to store the space dilation matrix and update it at every iteration, resulting in a substantial computational burden for large-sized problems. To circumvent this difficulty, we first propose a memoryless update scheme, which under a suitable choice of parameters, yields a direction of motion that turns out to be a convex combination of two successive anti-subgradients. Moreover, in the space transformation sense, the new update scheme can be viewed as a combination of space dilation and reduction operations. We prove convergence of this new method, and demonstrate how it can be used in conjunction with a variable target value method that allows a practical, convergent implementation of the method. We also examine a memoryless variant that uses a fixed dilation parameter instead of varying degrees of dilation and/or reduction as in the former algorithm, as well as another variant that examines a two-step limited memory update. These variants are tested along with Shor's r-algorithm and also a modified version of a related algorithm due to Polyak that employs a projection onto a pair of Kelley's cutting planes. We use a set of standard test problems from the literature as well as randomly generated dual transportation and assignment problems in our computational experiments. The results exhibit that the proposed space dilation and reduction method and the modification of Polyak's method are competitive, and offer a substantial advantage over the r-algorithm and over the other tested limited memory variants with respect to accuracy as well as effort.

[1]  E. Polak.  Introduction to linear and nonlinear programming , 1973 .

[2]  Boris Polyak.  Minimization of unsmooth functionals , 1969 .

[3]  N. Z. Shor.  Convergence of a gradient method with space dilation in the direction of the difference between two successive gradients , 1975 .

[4]  N. Z. Shor.  The rate of convergence of the generalized gradient descent method , 1968 .

[5]  J. E. Kelley,et al.  The Cutting-Plane Method for Solving Convex Programs , 1960 .

[6]  F. Clarke.  Generalized gradients and applications , 1975 .

[7]  C. Lemaréchal.  An extension of davidon methods to non differentiable problems , 1975 .

[8]  R. Tyrrell Rockafellar,et al.  A dual approach to solving nonlinear programming problems by unconstrained optimization , 1973, Math. Program..

[9]  N. Shor.  Nondifferentiable Optimization and Polynomial Problems , 1998 .

[10]  Jean-Louis Goffin,et al.  On convergence rates of subgradient optimization methods , 1977, Math. Program..

[11]  L. Nazareth.  A Relationship between the BFGS and Conjugate Gradient Algorithms and Its Implications for New Algorithms , 1979 .

[12]  N. Z. Shor.  Convergence rate of the gradient descent method with dilatation of the space , 1970 .

[13]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[14]  Hyunsil Ahn,et al.  Two-direction subgradient method for non-differentiable optimization problems , 1987 .

[15]  Philip Wolfe,et al.  Validation of subgradient optimization , 1974, Math. Program..

[16]  Roger Fletcher,et al.  A Rapidly Convergent Descent Method for Minimization , 1963, Comput. J..

[17]  H. Sherali,et al.  Conjugate gradient methods using quasi-Newton updates with inexact line searches , 1990 .

[18]  Krzysztof C. Kiwiel,et al.  Proximity control in bundle methods for convex nondifferentiable minimization , 1990, Math. Program..

[19]  N. Z. Shor.  Utilization of the operation of space dilatation in the minimization of convex functions , 1972 .

[20]  David G. Luenberger,et al.  Introduction to Linear and Nonlinear Programming , 1973 .

[21]  V. A. Skokov.  Note on minimization methods employing space stretching , 1974 .

[22]  C. Lemaréchal.  Constructing Bundle Methods for Convex Optimization , 1986 .

[23]  H. Sherali,et al.  A primal-dual conjugate subgradient algorithm for specially structured linear and convex programming problems , 1989 .

[24]  Naum Zuselevich Shor,et al.  Minimization Methods for Non-Differentiable Functions , 1985, Springer Series in Computational Mathematics.

[25]  E. A. Nurminskii,et al.  ∈-Quasigradient method for solving nonsmooth extremal problems , 1977, Cybernetics.

[26]  Hanif D. Sherali,et al.  A variable target value method for nondifferentiable optimization , 2000, Oper. Res. Lett..

[27]  J. Ben Rosen,et al.  Pracniques: construction of nonlinear programming test problems , 1965, Commun. ACM.

[28]  Gray De.  Bureau of standards. , 1989 .

[29]  P. Wolfe.  Note on a method of conjugate subgradients for minimizing nondifferentiable functions , 1974 .

[30]  C. Lemaréchal,et al.  Nonsmooth optimization : proceedings of a IIASA workshop, March 28-April 8, 1977 , 1978 .

[31]  Sehun Kim,et al.  An improved subgradient method for constrained nondifferentiable optimization , 1993, Oper. Res. Lett..

[32]  J. B. Rosen,et al.  Construction of nonlinear programming test problems , 1965 .

[33]  Philip Wolfe,et al.  Note on a method of conjugate subgradients for minimizing nondifferentiable functions , 1974, Math. Program..

[34]  C. M. Shetty,et al.  Nonlinear Programming - Theory and Algorithms, Second Edition , 1993 .

[35]  Mokhtar S. Bazaraa,et al.  Nonlinear Programming: Theory and Algorithms , 1993 .

[36]  Sehun Kim,et al.  Convergence of a generalized subgradient method for nondifferentiable convex optimization , 1991, Math. Program..

[37]  Hanif D. Sherali,et al.  Dual formulations and subgradient optimization strategies for linear programming relaxations of mixed-integer programs , 1988, Discret. Appl. Math..

[38]  Sehun Kim,et al.  Variable target value subgradient method , 1991, Math. Program..

[39]  P. Camerini,et al.  On improving relaxation methods by modified gradient techniques , 1975 .

[40]  N. Z. Shor,et al.  Solution of minimax problems by the method of generalized gradient descent with dilatation of the space , 1972 .

[41]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[42]  Krzysztof C. Kiwiel,et al.  A tilted cutting plane proximal bundle method for convex nondifferentiable optimization , 1991, Oper. Res. Lett..

[43]  Robert Mifflin,et al.  An Algorithm for Constrained Optimization with Semismooth Functions , 1977, Math. Oper. Res..

[44]  J. Goffin.  CONVERGENCE RESULTS IN A CLASS OF VARIABLE METRIC SUBGRADIENT METHODS , 1981 .

[45]  Jeffery L. Kennington,et al.  A generalization of Polyak's convergence result for subgradient optimization , 1987, Math. Program..