Monte Carlo Tree Search With Reinforcement Learning for Motion Planning

Motion planning for an autonomous vehicle is most challenging for scenarios such as large, multi-lane, and unsignalized intersections in the presence of dense traffic. In such situations, the motion planner has to deal with multiple crossing-points to reach an objective in a safe, comfortable, and efficient way. In addition, motion planning challenges include real-time computation and scalability to complex scenes with many objects and different road geometries. In this work, we propose a motion planning system addressing these challenges. We enable real-time applicability of a Monte Carlo Tree Search algorithm with a deep-learning heuristic. We learn a fast evaluation function from accurate, but non real-time models. While using Deep Reinforcement Learning techniques we maintain a clear separation between making predictions and making decisions. We reduce the complexity of the search model and benchmark the proposed agent against multiple methods: rules-based, MCTS, $A^{*}$ search, deep learning, and Model Predictive Control. We show that our agent outperforms these other agents in a variety of challenging scenarios, where we benchmark safety, comfort and efficiency metrics.

[1]  Carl-Johan Hoel,et al.  Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Driving , 2019, IEEE Transactions on Intelligent Vehicles.

[2]  Matthias Althoff,et al.  High-level Decision Making for Safe and Reasonable Autonomous Lane Changing using Reinforcement Learning , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[3]  Matthias Klusch,et al.  Hybrid Online POMDP Planning and Deep Reinforcement Learning for Safer Self-Driving Cars , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[4]  Lorenz T. Biegler,et al.  On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming , 2006, Math. Program..

[5]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[6]  Mykel J. Kochenderfer,et al.  Reinforcement Learning with Probabilistic Guarantees for Autonomous Driving , 2019, ArXiv.

[7]  David Janz,et al.  Learning to Drive in a Day , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[8]  Christof Büskens,et al.  Controlling an Autonomous Vehicle with Deep Reinforcement Learning , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[9]  Carl-Johan Hoel,et al.  Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[10]  David Hsu,et al.  LeTS-Drive: Driving in a Crowd by Learning from Tree Search , 2019, Robotics: Science and Systems.

[11]  Iain Dunning,et al.  JuMP: A Modeling Language for Mathematical Optimization , 2015, SIAM Rev..

[12]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[13]  Jonas Sjöberg,et al.  Learning When to Drive in Intersections by Combining Reinforcement Learning and Model Predictive Control , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[14]  Demis Hassabis,et al.  Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm , 2017, ArXiv.

[15]  Razvan Pascanu,et al.  Learning to Navigate in Complex Environments , 2016, ICLR.

[16]  Emilio Frazzoli,et al.  A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles , 2016, IEEE Transactions on Intelligent Vehicles.

[17]  Jonathan P. How,et al.  Decision Making Under Uncertainty: Theory and Application , 2015 .

[18]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.