Evaluating the Robustness of Deep Reinforcement Learning for Autonomous Policies in a Multi-Agent Urban Driving Environment

Background: Deep reinforcement learning is actively used for training autonomous car policies in a simulated driving environment. Due to the large availability of various reinforcement learning algorithms and the lack of their systematic comparison across different driving scenarios, we are unsure of which ones are more effective for training autonomous car software in single-agent as well as multi-agent driving environments. Aims: A benchmarking framework for the comparison of deep reinforcement learning in a vision-based autonomous driving will open up the possibilities for training better autonomous car driving policies. Method: To address these challenges, we provide an open and reusable benchmarking framework for systematic evaluation and comparative analysis of deep reinforcement learning algorithms for autonomous driving in a single- and multi-agent environment. Using the framework, we perform a comparative study of four discrete and two continuous action space deep reinforcement learning algorithms. We also propose a comprehensive multi-objective reward function designed for the evaluation of deep reinforcement learning-based autonomous driving agents. We run the experiments in a vision-only high-fidelity urban driving simulated environments. Results: The results indicate that only some of the deep reinforcement learning algorithms perform consistently better across single and multi-agent scenarios when trained in various multi-agent-only environment settings. For example, A3C- and TD3-based autonomous cars perform comparatively better in terms of more robust actions and minimal driving errors in both single and multi-agent scenarios. Conclusions: We conclude that different deep reinforcement learning algorithms exhibit different driving and testing performance in different scenarios, which underlines the need for their systematic comparative analysis. The benchmarking framework proposed in this paper facilitates such a comparison.

[1]  Bolei Zhou,et al.  MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Jingda Wu,et al.  Efficient Deep Reinforcement Learning With Imitative Expert Priors for Autonomous Driving , 2021, IEEE Transactions on Neural Networks and Learning Systems.

[3]  Haochen Liu,et al.  Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving , 2021, 2022 IEEE Intelligent Vehicles Symposium (IV).

[4]  Masayoshi Tomizuka,et al.  Interpretable End-to-End Urban Autonomous Driving With Latent Deep Reinforcement Learning , 2020, IEEE Transactions on Intelligent Transportation Systems.

[5]  Kyung-Joong Kim,et al.  A New Open-Source Off-road Environment for Benchmark Generalization of Autonomous Driving , 2021, IEEE Access.

[6]  Eric Sax,et al.  Evaluation of Deep Reinforcement Learning Algorithms for Autonomous Driving , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[7]  Paolo Tonella,et al.  Testing machine learning based systems: a systematic mapping , 2020, Empirical Software Engineering.

[8]  Bálint Gyires-Tóth,et al.  Robust Reinforcement Learning-based Autonomous Driving Agent for Simulation and Real World , 2020, 2020 International Joint Conference on Neural Networks (IJCNN).

[9]  Joshua Garcia,et al.  A Comprehensive Study of Autonomous Vehicle Bugs , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[10]  Dusica Marijan,et al.  Software Testing for Machine Learning , 2020, AAAI.

[11]  Rupsa Saha,et al.  Road Detection for Reinforcement Learning Based Autonomous Car , 2020, ICISS.

[12]  Piotr Milos,et al.  Simulation-Based Reinforcement Learning for Real-World Autonomous Driving , 2019, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[13]  Praveen Palanisamy,et al.  Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning , 2019, 2020 International Joint Conference on Neural Networks (IJCNN).

[14]  Nhat M. Nguyen,et al.  SMARTS: An Open-Source Scalable Multi-Agent RL Training School for Autonomous Driving , 2020, CoRL.

[15]  Javier Alonso-Mora,et al.  Social behavior for autonomous vehicles , 2019, Proceedings of the National Academy of Sciences.

[16]  Filippos Christianos,et al.  Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning , 2019, ArXiv.

[17]  Masayoshi Tomizuka,et al.  Model-free Deep Reinforcement Learning for Urban Autonomous Driving , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[18]  Behdad Chalaki,et al.  Simulation to scaled city: zero-shot policy transfer for traffic control via autonomous vehicles , 2018, ICCPS.

[19]  David Janz,et al.  Learning to Drive in a Day , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[20]  Rahul Mangharam,et al.  F1TENTH: An Open-source Evaluation Environment for Continuous Control and Reinforcement Learning , 2019, NeurIPS.

[21]  Sen Wang,et al.  Deep Reinforcement Learning for Autonomous Driving , 2018, ArXiv.

[22]  Paul Newman,et al.  Imminent Collision Mitigation with Reinforcement Learning and Vision , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[23]  Alexandre M. Bayen,et al.  Benchmarks for reinforcement learning in mixed-autonomy traffic , 2018, CoRL.

[24]  Sabir Hossain,et al.  Driverless Car: Autonomous Driving Using Deep Reinforcement Learning in Urban Environment , 2018, 2018 15th International Conference on Ubiquitous Robots (UR).

[25]  Fawzi Nashashibi,et al.  End-to-End Race Driving with Deep Reinforcement Learning , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[26]  Herke van Hoof,et al.  Addressing Function Approximation Error in Actor-Critic Methods , 2018, ICML.

[27]  Shane Legg,et al.  IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures , 2018, ICML.

[28]  Bowen Tan,et al.  Autonomous Driving in Reality with Reinforcement Learning and Image Translation , 2018, ArXiv.

[29]  Michael I. Jordan,et al.  Ray: A Distributed Framework for Emerging AI Applications , 2017, OSDI.

[30]  Max Jaderberg,et al.  Population Based Training of Neural Networks , 2017, ArXiv.

[31]  Germán Ros,et al.  CARLA: An Open Urban Driving Simulator , 2017, CoRL.

[32]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[33]  Johann Marius Zöllner,et al.  Learning how to drive in a real world simulation with deep Q-Networks , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[34]  Cewu Lu,et al.  Virtual to Real Reinforcement Learning for Autonomous Driving , 2017, BMVC.

[35]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[36]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[37]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[38]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[39]  Frans A. Oliehoek,et al.  Decentralized POMDPs , 2012, Reinforcement Learning.

[40]  Christos Dimitrakakis,et al.  TORCS, The Open Racing Car Simulator , 2005 .

[41]  Jeff G. Schneider,et al.  Approximate solutions for partially observable stochastic games with common payoffs , 2004, Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, 2004. AAMAS 2004..

[42]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.