An Integer Programming Approach to Optimal Derivative Accumulation

In automatic differentiation, vertex elimination is one of the many methods for Jacobian accumulation and in general it can be much more efficient than the forward mode or reverse mode (Forth et al. ACM Trans Math Softw 30(3):266–299, 2004; Griewank and Walther, Evaluating derivatives: principles and techniques of algorithmic differentiation, SIAM, Philadelphia, 2008). However, finding the optimal vertex elimination sequence of a computational graph is a hard combinatorial optimization problem. In this paper, we propose to tackle this problem with an integer programming (IP) technique, and we develop an IP formulation for it. This enables us to use a standard integer optimization solver to find an optimal vertex elimination strategy. In addition, we have developed several bound-tightening and symmetry-breaking constraints to strengthen the basic IP formulation. We demonstrate the effectiveness of these enhancements through computational experiments.

[1]  William J. Cook,et al.  The Traveling Salesman Problem: A Computational Study , 2007 .

[2]  Martin Berggren,et al.  Hybrid differentiation strategies for simulation and analysis of applications in C++ , 2008, TOMS.

[3]  Jacques F. Benders,et al.  Partitioning procedures for solving mixed-variables programming problems , 2005, Comput. Manag. Sci..

[4]  R. A. Zemlin,et al.  Integer Programming Formulation of Traveling Salesman Problems , 1960, JACM.

[5]  Heather Fry,et al.  A user’s guide , 2003 .

[6]  A. Griewank,et al.  On the calculation of Jacobian matrices by the Markowitz rule , 1991 .

[7]  Andreas Griewank,et al.  Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition , 2000, Frontiers in applied mathematics.

[8]  Uwe Naumann,et al.  On optimality preserving eliminations for the minimum edge count and optimal Jacobian accumulation problems in linearized DAGs , 2012, Optim. Methods Softw..

[9]  John D. Pryce,et al.  Jacobian code generated by source transformation and vertex elimination can be as efficient as hand-coding , 2004, TOMS.

[10]  Yuxiao Hu,et al.  Optimal vertex elimination in single-expression-use graphs , 2008, TOMS.

[11]  Richard E. Rosenthal,et al.  GAMS -- A User's Guide , 2004 .

[12]  Uwe Naumann,et al.  Optimal Jacobian accumulation is NP-complete , 2007, Math. Program..

[13]  Laurence A. Wolsey,et al.  Integer and Combinatorial Optimization , 1988 .

[14]  Uwe Naumann,et al.  Simulated Annealing for Optimal Pivot Selection in Jacobian Accumulation , 2003, SAGA.

[15]  Uwe Naumann,et al.  Markowitz-Type Heuristics for Computing Jacobian Matrices Efficiently , 2003, International Conference on Computational Science.

[16]  William J. Cook,et al.  The Traveling Salesman Problem: A Computational Study (Princeton Series in Applied Mathematics) , 2007 .