A vector forward mode of automatic differentiation for generalized derivative evaluation

Numerical methods for non-smooth equation-solving and optimization often require generalized derivative information in the form of elements of the Clarke Jacobian or the B-subdifferential. It is shown here that piecewise differentiable functions are lexicographically smooth in the sense of Nesterov, and that lexicographic derivatives of these functions comprise a particular subset of both the B-subdifferential and the Clarke Jacobian. Several recently developed methods for generalized derivative evaluation of composite piecewise differentiable functions are shown to produce identical results, which are also lexicographic derivatives. A vector forward mode of automatic differentiation (AD) is presented for evaluation of these derivatives, generalizing established methods and combining their computational benefits. This forward AD mode may be applied to any finite composition of known smooth functions, piecewise differentiable functions such as the absolute value function, , and , and certain non-smooth functions which are not piecewise differentiable, such as the Euclidean norm. This forward AD mode may be implemented using operator overloading, does not require storage of a computational graph, and is computationally tractable relative to the cost of a function evaluation. An implementation in C is discussed.

[1]  Michael Ulbrich,et al.  Semismooth Newton Methods for Variational Inequalities and Constrained Optimization Problems in Function Spaces , 2011, MOS-SIAM Series on Optimization.

[2]  Daniel Ralph,et al.  Global Convergence of Damped Newton's Method for Nonsmooth Equations via the Path Search , 1994, Math. Oper. Res..

[3]  Andreas Griewank,et al.  On stable piecewise linearization and generalized algorithmic differentiation , 2013, Optim. Methods Softw..

[4]  K. Kiwiel.  Methods of Descent for Nondifferentiable Optimization , 1985 .

[5]  F. Facchinei,et al.  Finite-Dimensional Variational Inequalities and Complementarity Problems , 2003 .

[6]  Michael Hintermüller,et al.  A SQP-Semismooth Newton-type Algorithm applied to Control of the instationary Navier--Stokes System Subject to Control Constraints , 2006, SIAM J. Optim..

[7]  F. Clarke.  Optimization And Nonsmooth Analysis , 1983 .

[8]  Paul I. Barton,et al.  Evaluating an element of the Clarke generalized Jacobian of a composite piecewise differentiable function , 2013, TOMS.

[9]  R. Mifflin.  Semismooth and Semiconvex Functions in Constrained Optimization , 1977 .

[10]  Xiaojun Chen,et al.  Smoothing Methods and Semismooth Methods for Nondifferentiable Operator Equations , 2000, SIAM J. Numer. Anal..

[11]  Yurii Nesterov,et al.  Lexicographic differentiation of nonsmooth functions , 2005, Math. Program..

[12]  D. Klatte.  Nonsmooth equations in optimization , 2002 .

[13]  S. Dirkse,et al.  The path solver: a nommonotone stabilization scheme for mixed complementarity problems , 1995 .

[14]  Jirí V. Outrata,et al.  Shape Optimization in Three-Dimensional Contact Problems with Coulomb Friction , 2009, SIAM J. Optim..

[15]  Bangti Jin,et al.  A Semismooth Newton Method for L1 Data Fitting with Automatic Choice of Regularization Parameters and Noise Calibration , 2010, SIAM J. Imaging Sci..

[16]  Zdzisław Denkowski,et al.  Set-Valued Analysis , 2021 .

[17]  B. Mordukhovich.  Generalized Differential Calculus for Nonsmooth and Set-Valued Mappings , 1994 .

[18]  T. H. III Sweetser,et al.  A minimal set-valued strong derivative for vector-valued Lipschitz functions , 1977 .

[19]  Jong-Shi Pang,et al.  NE/SQP: A robust algorithm for the nonlinear complementarity problem , 1993, Math. Program..

[20]  Jan Vlcek,et al.  Algorithm 811: NDA: algorithms for nondifferentiable optimization , 2001, TOMS.

[21]  Jochem Zowe,et al.  A Version of the Bundle Idea for Minimizing a Nonsmooth Function: Conceptual Idea, Convergence Analysis, Numerical Results , 1992, SIAM J. Optim..

[22]  S. Scholtes.  Introduction to Piecewise Differentiable Equations , 2012 .

[23]  Huifu Xu,et al.  Set-valued approximations and Newton’s methods , 1999, Math. Program..

[24]  Liqun Qi,et al.  Convergence Analysis of Some Algorithms for Solving Nonsmooth Equations , 1993, Math. Oper. Res..

[25]  José Herskovits,et al.  Globally Convergent Cutting Plane Method for Nonconvex Nonsmooth Minimization , 2011, J. Optim. Theory Appl..

[26]  Michael Ulbrich,et al.  Semismooth Newton Methods for Operator Equations in Function Spaces , 2002, SIAM J. Optim..

[27]  P. Neittaanmäki,et al.  Nonsmooth Optimization: Analysis and Algorithms with Applications to Optimal Control , 1992 .

[28]  Paul I. Barton,et al.  Generalized Derivatives for Solutions of Parametric Ordinary Differential Equations with Non-differentiable Right-Hand Sides , 2014, J. Optim. Theory Appl..

[29]  M. Kojima,et al.  EXTENSION OF NEWTON AND QUASI-NEWTON METHODS TO SYSTEMS OF PC^1 EQUATIONS , 1986 .

[30]  Marko Mäkelä,et al.  Survey of Bundle Methods for Nonsmooth Optimization , 2002, Optim. Methods Softw..

[31]  Jiří V. Outrata,et al.  Shape Optimization in 2D Contact Problems with Given Friction and a Solution-Dependent Coefficient of Friction , 2012 .

[32]  Francisco Facchinei,et al.  An LP-Newton method: nonsmooth equations, KKT systems, and nonisolated solutions , 2013, Mathematical Programming.

[33]  Franziska Wulf,et al.  Minimization Methods For Non Differentiable Functions , 2016 .

[34]  A. Griewank.  Automatic Directional Differentiation of Nonsmooth Composite Functions , 1995 .

[35]  Andreas Griewank,et al.  Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition , 2000, Frontiers in applied mathematics.

[36]  J. Hiriart-Urruty,et al.  Convex analysis and minimization algorithms , 1993 .

[37]  Kazufumi Ito,et al.  The Primal-Dual Active Set Strategy as a Semismooth Newton Method , 2002, SIAM J. Optim..

[38]  Georg Stadler,et al.  Elliptic optimal control problems with L1-control cost and applications for the placement of control devices , 2009, Comput. Optim. Appl..

[39]  Michael C. Ferris,et al.  Interfaces to PATH 3.0: Design, Implementation and Usage , 1999, Comput. Optim. Appl..

[40]  C. Lemaréchal,et al.  ON A BUNDLE ALGORITHM FOR NONSMOOTH OPTIMIZATION , 1981 .

[41]  Liqun Qi,et al.  A nonsmooth version of Newton's method , 1993, Math. Program..