d(Tree)-by-dx: Automatic and Exact Differentiation of Genetic Programming Trees

Genetic programming (GP) has developed to the point where it is a credible candidate for the ‘black box’ modeling of real systems. Wider application, however, could greatly benefit from its seamless embedding in conventional optimization schemes, which are most efficiently carried out using gradient-based methods. This paper describes the development of a method to automatically differentiate GP trees using a series of tree transformation rules; the resulting method can be applied an unlimited number of times to obtain higher derivatives of the function approximated by the original, trained GP tree. We demonstrate the utility of our method using a number of illustrative gradient-based optimizations that embed GP models.

[1]  E. A. Hathway,et al.  Model-predictive control for non-domestic buildings: a critical review and prospects , 2017 .

[2]  Dieter Kraft,et al.  Algorithm 733: TOMP–Fortran modules for optimal control calculations , 1994, TOMS.

[3]  Lorenz T. Biegler,et al.  On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming , 2006, Math. Program..

[4]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[5]  Klaus Schittkowski,et al.  More test examples for nonlinear programming codes , 1981 .

[6]  Maarten Keijzer,et al.  Improving Symbolic Regression with Interval Arithmetic and Linear Scaling , 2003, EuroGP.

[7]  W. Rudin Principles of mathematical analysis , 1964 .

[8]  Katya Scheinberg,et al.  Introduction to derivative-free optimization , 2010, Math. Comput..

[9]  Barak A. Pearlmutter,et al.  Automatic differentiation in machine learning: a survey , 2015, J. Mach. Learn. Res..

[10]  Dario Izzo,et al.  Differentiable Genetic Programming , 2016, EuroGP.

[11]  L. Dixon,et al.  Automatic differentiation of algorithms , 2000 .

[12]  Jorge Nocedal,et al.  An interior algorithm for nonlinear optimization that combines line search and trust region steps , 2006, Math. Program..

[13]  Samaneh Sadat Mousavi Astarabadi,et al.  Avoiding Overfitting in Symbolic Regression Using the First Order Derivative of GP Trees , 2015, GECCO.

[14]  Conor Ryan,et al.  Handbook of Genetic Programming Applications , 2015, Springer International Publishing.

[15]  Peter Rockett,et al.  The Use of an Analytic Quotient Operator in Genetic Programming , 2013, IEEE Transactions on Evolutionary Computation.