Improved Policy Extraction via Online Q-Value Distillation

Deep neural networks are capable of solving complex control tasks in challenging environments, but their learned policies are hard to interpret. Not being able to explain or verify them limits their practical applicability. By contrast, decision trees lend themselves well to explanation and verification, but are not easy to train, especially in an online fashion. In this work we introduce Q-BSP trees and propose an Ordered Sequential Monte Carlo training algorithm that efficiently distills the Q-function from fully trained deep Q-networks into a tree structure. Q-BSP forests are used to generate the partitioning rules that transparently reconstruct an accurate value function. We explain our approach and provide results that convincingly beat earlier online policy distillation methods with respect to their own performance benchmarks.

[1]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[2]  Rich Caruana,et al.  Do Deep Nets Really Need to be Deep? , 2013, NIPS.

[3]  Bin Li,et al.  The Binary Space Partitioning-Tree Process , 2018, AISTATS.

[4]  Olcay Boz,et al.  Extracting decision trees from trained neural networks , 2002, KDD.

[5]  Rich Caruana,et al.  Model compression , 2006, KDD '06.

[6]  Saso Dzeroski,et al.  Learning model trees from evolving data streams , 2010, Data Mining and Knowledge Discovery.

[7]  Ankur Taly,et al.  Axiomatic Attribution for Deep Networks , 2017, ICML.

[8]  Yee Whye Teh,et al.  The Mondrian Process , 2008, NIPS.

[9]  Pierre Geurts,et al.  Tree-Based Batch Mode Reinforcement Learning , 2005, J. Mach. Learn. Res..

[10]  Yan Liu,et al.  Interpretable Deep Models for ICU Outcome Prediction , 2016, AMIA.

[11]  J. R. Quinlan.  Learning With Continuous Classes , 1992 .

[12]  Antonio Criminisi,et al.  Measuring Neural Net Robustness with Constraints , 2016, NIPS.

[13]  Pierre Geurts,et al.  Extremely randomized trees , 2006, Machine Learning.

[14]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[15]  Pieter Abbeel,et al.  Apprenticeship learning via inverse reinforcement learning , 2004, ICML.

[16]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[17]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[18]  Andrew W. Moore,et al.  Efficient memory-based learning for robot control , 1990 .

[19]  Richard S. Sutton,et al.  Neuronlike adaptive elements that can solve difficult learning control problems , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[20]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[21]  Oliver Schulte,et al.  Toward Interpretable Deep Reinforcement Learning with Linear Model U-Trees , 2018, ECML/PKDD.

[22]  David McLean,et al.  Logistic Model Tree Extraction From Artificial Neural Networks , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[23]  Sung-Hyun Son,et al.  Interpretable Reinforcement Learning via Differentiable Decision Trees , 2019, ArXiv.

[24]  Yee Whye Teh,et al.  Mondrian Forests: Efficient Online Random Forests , 2014, NIPS.

[25]  H. Chipman,et al.  BART: Bayesian Additive Regression Trees , 2008, 0806.3286.

[26]  Eduardo F. Morales,et al.  An Introduction to Reinforcement Learning , 2011 .

[27]  Paul Pettersson,et al.  Tools and Algorithms for the Construction and Analysis of Systems: 28th International Conference, TACAS 2022, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2022, Munich, Germany, April 2–7, 2022, Proceedings, Part II , 1998, TACAS.

[28]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[29]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[30]  Armando Solar-Lezama,et al.  Verifiable Reinforcement Learning via Policy Extraction , 2018, NeurIPS.

[31]  Filip De Turck,et al.  GENESIM: genetic extraction of a single, interpretable model , 2016, NIPS 2016.

[32]  John Schulman,et al.  Concrete Problems in AI Safety , 2016, ArXiv.

[33]  Andreas Krause,et al.  Safe Model-based Reinforcement Learning with Stability Guarantees , 2017, NIPS.

[34]  Bin Li,et al.  Binary Space Partitioning Forests , 2019, ArXiv.