Cheaper adjoints by reversing address computations

The reverse mode of automatic differentiation is widely used in science and engineering. A severe bottleneck for the performance of the reverse mode, however, is the necessity to recover certain intermediate values of the program in reverse order. Among these values are computed addresses, which traditionally are recovered through forward recomputation and storage in memory. We propose an alternative approach for recovery that uses inverse computation based on dependency information. Address storage constitutes a significant portion of the overall storage requirements. An example illustrates substantial gains that the proposed approach yields, and we show use cases in practical applications.

[1]  Uwe Naumann,et al.  "To be recorded" analysis in reverse-mode automatic differentiation , 2005, Future Gener. Comput. Syst..

[2]  Dafydd Gibbon,et al.  1 User’s guide , 1998 .

[3]  Trond Steihaug,et al.  Truncated-newtono algorithms for large-scale unconstrained optimization , 1983, Math. Program..

[4]  Richard J. Fateman,et al.  Automatic Differentiation of Algorithms: Theory, Implementation, and Application (Andreas Griewank and George F. Corliss, eds.) , 1993, SIAM Rev..

[5]  Thomas Kaminski,et al.  Recomputations in reverse mode AD , 2000 .

[6]  Antony Jameson,et al.  Aerodynamic design via control theory , 1988, J. Sci. Comput..

[7]  Jean Utke,et al.  OpenAD/F: A Modular Open-Source Tool for Automatic Differentiation of Fortran Codes , 2008, TOMS.

[8]  Uwe Naumann,et al.  Automatic Differentiation: Applications, Theory, and Implementations (Lecture Notes in Computational Science and Engineering) , 2006 .

[9]  F. L. Dimet,et al.  Variational algorithms for analysis and assimilation of meteorological observations: theoretical aspects , 1986 .

[10]  Andreas Griewank,et al.  Automatic Differentiation of Algorithms: From Simulation to Optimization , 2000, Springer New York.

[11]  Monica S. Lam,et al.  A Loop Transformation Theory and an Algorithm to Maximize Parallelism , 1991, IEEE Trans. Parallel Distributed Syst..

[12]  Uwe Naumann.  On Optimal DAG Reversal , 2007 .

[13]  Andreas Griewank,et al.  Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition , 2000, Frontiers in applied mathematics.

[14]  Laurent Hascoët,et al.  TAPENADE 2.1 user's guide , 2004 .

[15]  Andreas Griewank,et al.  A mathematical view of automatic differentiation , 2003, Acta Numerica.

[16]  Steven S. Muchnick,et al.  Advanced Compiler Design and Implementation , 1997 .

[17]  Michael Wolfe,et al.  Beyond induction variables , 1992, PLDI '92.

[18]  Uwe Naumann,et al.  Minimizing the tape size , 2000 .

[19]  Martin Berz,et al.  Computational differentiation : techniques, applications, and tools , 1996 .

[20]  Michael Griebel,et al.  Numerical Simulation in Fluid Dynamics: A Practical Introduction , 1997 .

[21]  Ken Kennedy,et al.  Automatic translation of FORTRAN programs to vector form , 1987, TOPL.