Branch-locking AD techniques for nonsmooth composite functions and nonsmooth implicit functions

A recent nonsmooth vector forward mode of algorithmic differentiation (AD) computes Nesterov's L-derivatives for nonsmooth composite functions; these L-derivatives provide useful sensitivity information to methods for nonsmooth optimization and equation solving. The established reverse AD mode evaluates gradients efficiently for smooth functions, but it does not extend directly to nonsmooth functions. Thus, this article examines branch-locking strategies to harness the benefits of smooth AD techniques even in the nonsmooth case, in order to improve the computational performance of the nonsmooth vector forward AD mode. In these strategies, each nonsmooth elemental function in the original composition is ‘locked’ into an appropriate linear ‘branch’. The original composition is thereby replaced with a smooth variant, which may be subjected to efficient AD techniques for smooth functions such as the reverse AD mode. In order to choose the correct linear branches, we use inexpensive probing steps to ascertain the composite function's local behaviour. A simple implementation in is described, and the developed techniques are extended to nonsmooth local implicit functions and inverse functions.

[1]  Andreas Griewank,et al.  On Lipschitz optimization based on gray-box piecewise linearization , 2016, Math. Program..

[2]  Paul I. Barton,et al.  Generalized Derivatives for Solutions of Parametric Ordinary Differential Equations with Non-differentiable Right-Hand Sides , 2014, J. Optim. Theory Appl..

[3]  Ignacio E. Grossmann,et al.  Simultaneous optimization and heat integration of chemical processes , 1986 .

[4]  M. Kojima,et al.  EXTENSION OF NEWTON AND QUASI-NEWTON METHODS TO SYSTEMS OF PC^1 EQUATIONS , 1986 .

[5]  T. H. III Sweetser,et al.  A minimal set-valued strong derivative for vector-valued Lipschitz functions , 1977 .

[6]  K. Kiwiel.  Methods of Descent for Nondifferentiable Optimization , 1985 .

[7]  R. Raman,et al.  Modelling and computational techniques for logic based integer programming , 1994 .

[8]  A. Griewank.  Automatic Directional Differentiation of Nonsmooth Composite Functions , 1995 .

[9]  Yurii Nesterov,et al.  Lexicographic differentiation of nonsmooth functions , 2005, Math. Program..

[10]  Paul I. Barton,et al.  Generalized Derivatives for Hybrid Systems , 2017, IEEE Transactions on Automatic Control.

[11]  Andreas Griewank,et al.  Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition , 2000, Frontiers in applied mathematics.

[12]  Paul I. Barton,et al.  A vector forward mode of automatic differentiation for generalized derivative evaluation , 2015, Optim. Methods Softw..

[13]  Napsu Karmitsa,et al.  Test problems for large-scale nonsmooth minimization , 2007 .

[14]  Liqun Qi,et al.  A nonsmooth version of Newton's method , 1993, Math. Program..

[15]  Paul I. Barton,et al.  Multistream heat exchanger modeling and design , 2015 .

[16]  Ramon E. Moore.  Methods and applications of interval analysis , 1979, SIAM studies in applied mathematics.

[17]  P I Barton,et al.  A reliable simulator for dynamic flux balance analysis , 2013, Biotechnology and bioengineering.

[18]  H. Nijmeijer,et al.  Dynamics and Bifurcations ofNon - Smooth Mechanical Systems , 2006 .

[19]  John N. Tsitsiklis,et al.  Introduction to linear optimization , 1997, Athena scientific optimization and computation series.

[20]  M A Henson,et al.  Steady-state and dynamic flux balance analysis of ethanol production by Saccharomyces cerevisiae. , 2009, IET systems biology.

[21]  Stephen F. Siegel,et al.  FEVS: A Functional Equivalence Verification Suite for High-Performance Scientific Computing , 2011, Math. Comput. Sci..

[22]  A. Neumaier.  Interval methods for systems of equations , 1990 .

[23]  Michael Ulbrich,et al.  Semismooth Newton Methods for Variational Inequalities and Constrained Optimization Problems in Function Spaces , 2011, MOS-SIAM Series on Optimization.

[24]  Andreas Griewank,et al.  On stable piecewise linearization and generalized algorithmic differentiation , 2013, Optim. Methods Softw..

[25]  A. Griewank,et al.  Solving piecewise linear systems in abs-normal form , 2015, 1701.00753.

[26]  F. Facchinei,et al.  Finite-Dimensional Variational Inequalities and Complementarity Problems , 2003 .

[27]  Ian A. Hiskens,et al.  Stability of hybrid system limit cycles: application to the compass gait biped robot , 2001, Proceedings of the 40th IEEE Conference on Decision and Control (Cat. No.01CH37228).

[28]  S. Scholtes.  Introduction to Piecewise Differentiable Equations , 2012 .

[29]  Sven Leyffer,et al.  Solving mixed integer nonlinear programs by outer approximation , 1994, Math. Program..

[30]  F. Clarke.  Optimization And Nonsmooth Analysis , 1983 .

[31]  Paul I. Barton,et al.  Generalized Derivatives of Differential–Algebraic Equations , 2016, J. Optim. Theory Appl..

[32]  Paul I. Barton,et al.  Nonsmooth model for dynamic simulation of phase changes , 2016 .

[33]  Richard W. Cottle,et al.  Linear Complementarity Problem , 2009, Encyclopedia of Optimization.

[34]  Paul I. Barton,et al.  Evaluating an element of the Clarke generalized Jacobian of a composite piecewise differentiable function , 2013, TOMS.

[35]  Paul I. Barton,et al.  Generalized derivatives of dynamic systems with a linear program embedded , 2016, Autom..

[36]  Paul I. Barton,et al.  McCormick-Based Relaxations of Algorithms , 2009, SIAM J. Optim..

[37]  Paul I. Barton,et al.  Computationally relevant generalized derivatives: theory, evaluation and applications , 2018, Optim. Methods Softw..

[38]  Lorenz T. Biegler,et al.  MPEC strategies for cost optimization of pipeline operations , 2010, Comput. Chem. Eng..

[39]  Uwe Naumann,et al.  The Art of Differentiating Computer Programs - An Introduction to Algorithmic Differentiation , 2012, Software, environments, tools.

[40]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[41]  Stephen F. Siegel,et al.  The Toolkit for Accurate Scientific Software , 2011 .

[42]  U. Naumann,et al.  Adjoint Mode Computation of Subgradients for McCormick Relaxations , 2012 .

[43]  Stephen F. Siegel,et al.  TASS: The Toolkit for Accurate Scientific Software , 2011, Math. Comput. Sci..

[44]  Garth P. McCormick,et al.  Computability of global solutions to factorable nonconvex programs: Part I — Convex underestimating problems , 1976, Math. Program..

[45]  Paul I. Barton,et al.  Generalized Derivatives of Lexicographic Linear Programs , 2018, J. Optim. Theory Appl..

[46]  Liqun Qi,et al.  Convergence Analysis of Some Algorithms for Solving Nonsmooth Equations , 1993, Math. Oper. Res..

[47]  Francisco Facchinei,et al.  An LP-Newton method: nonsmooth equations, KKT systems, and nonisolated solutions , 2013, Mathematical Programming.

[48]  Franziska Wulf,et al.  Minimization Methods For Non Differentiable Functions , 2016 .