A Hierarchical Framework for Multi-Lane Autonomous Driving Based on Reinforcement Learning

This paper proposes a hierarchical framework integrating deep reinforcement learning (DRL) and rule-based methods for multi-lane autonomous driving. We define an instantaneous desired speed (IDS) to mimic the common motivation for higher speed in different traffic situations as an intermediate action. High-level DRL is utilized to generate IDS directly, while the low-level rule-based policies including car following (CF) models and three-stage lane changing (LC) models are governed by the common goal of IDS. Not only the coupling between CF and LC behaviors is captured by the hierarchy, but also utilizing the benefits from both DRL and rule-based methods like more interpretability and learning ability. Owing to the decomposition and combination with rule-based models, traffic flow operations can be taken into account for individually controlled automated vehicles (AVs) with an extension of traffic flow adaptive (TFA) strategy through exposed critical parameters. A comprehensive evaluation for the proposed framework is conducted from both the individual and system perspective, comparing with a pure DRL model and widely used rule-based model IDM with MOBIL. The simulation results prove the effectiveness of the proposed framework.

[1]  Danjue Chen,et al.  A deep reinforcement learning based distributed control strategy for connected automated vehicles in mixed traffic platoon , 2023, Transportation Research Part C: Emerging Technologies.

[2]  Zhibin Li,et al.  An Integrated Model for Autonomous Speed and Lane Change Decision-Making Based on Deep Reinforcement Learning , 2022, IEEE Transactions on Intelligent Transportation Systems.

[3]  Yuanchang Xie,et al.  Reinforcement Learning based cooperative longitudinal control for reducing traffic oscillations and improving platoon stability , 2022, Transportation Research Part C: Emerging Technologies.

[4]  Diange Yang,et al.  Trustworthy safety improvement for autonomous driving using reinforcement learning , 2022, Transportation Research Part C: Emerging Technologies.

[5]  B. Ran,et al.  Connected automated vehicle cooperative control with a deep reinforcement learning approach in a mixed traffic environment , 2021, Transportation Research Part C: Emerging Technologies.

[6]  Bo Zou,et al.  Deep reinforcement learning in transportation research: A review , 2021 .

[7]  Marcello Montanino,et al.  About calibration of car-following dynamics of automated and human-driven vehicles: Methodology, guidelines and codes , 2021, Transportation Research Part C: Emerging Technologies.

[8]  Zhiheng Li,et al.  Harmonious Lane Changing via Deep Reinforcement Learning , 2021, IEEE Transactions on Intelligent Transportation Systems.

[9]  Xuan Di,et al.  A Physics-Informed Deep Learning Paradigm for Car-Following Models , 2020, ArXiv.

[10]  Marcello Restelli,et al.  Combining reinforcement learning with rule-based controllers for transparent and general decision-making in autonomous driving , 2020, Robotics Auton. Syst..

[11]  Victor Talpaert,et al.  Deep Reinforcement Learning for Autonomous Driving: A Survey , 2020, IEEE Transactions on Intelligent Transportation Systems.

[12]  Qing Tang,et al.  Exploration Entropy for Reinforcement Learning , 2020 .

[13]  Sergey Levine,et al.  Planning with Goal-Conditioned Policies , 2019, NeurIPS.

[14]  Nazim Kemal Ure,et al.  Automated Lane Change Decision Making using Deep Reinforcement Learning in Dynamic and Uncertain Highway Environment , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[15]  Bin Jia,et al.  A data-driven lane-changing model based on deep learning , 2019, Transportation Research Part C: Emerging Technologies.

[16]  Xiao Qi,et al.  Simultaneous modeling of car-following and lane-changing behaviors using deep learning , 2019, Transportation Research Part C: Emerging Technologies.

[17]  Alexander Carballo,et al.  A Survey of Autonomous Driving: Common Practices and Emerging Technologies , 2019, IEEE Access.

[18]  Sergey Levine,et al.  Causal Confusion in Imitation Learning , 2019, NeurIPS.

[19]  Prashant Doshi,et al.  Merging in Congested Freeway Traffic Using Multipolicy Decision Making and Passive Actor-Critic Learning , 2019, IEEE Transactions on Intelligent Vehicles.

[20]  Meixin Zhu,et al.  Human-Like Autonomous Car-Following Model with Deep Reinforcement Learning , 2018, Transportation Research Part C: Emerging Technologies.

[21]  Cynthia Rudin,et al.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead , 2018, Nature Machine Intelligence.

[22]  Alexandre M. Bayen,et al.  Lagrangian Control through Deep-RL: Applications to Bottleneck Decongestion , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[23]  Fangyu Wu,et al.  Connections between classical car following models and artificial neural networks , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[24]  Alexandre M. Bayen,et al.  Dissipating stop-and-go waves in closed and open networks via deep reinforcement learning , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[25]  Qin Lin,et al.  Lane-Change Intention Estimation for Car-Following Control in Autonomous Driving , 2018, IEEE Transactions on Intelligent Vehicles.

[26]  K. Madhava Krishna,et al.  Overtaking Maneuvers in Simulated Highway Driving using Deep Reinforcement Learning , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[27]  Alexandre M. Bayen,et al.  Stabilizing Traffic with Autonomous Vehicles , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[28]  Ching-Yao Chan,et al.  A Reinforcement Learning Based Approach for Automated Lane Change Maneuvers , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[29]  Xiaohui Zhang,et al.  Automated vehicle's behavior decision making using deep reinforcement learning and high-fidelity simulation environment , 2018, Transportation Research Part C: Emerging Technologies.

[30]  Jian Sun,et al.  Stability analysis methods and their applicability to car-following models in conventional and connected environments , 2018 .

[31]  Alexandre M. Bayen,et al.  Emergent Behaviors in Mixed-Autonomy Traffic , 2017, CoRL.

[32]  Alexandre M. Bayen,et al.  Flow: A Modular Learning Framework for Mixed Autonomy Traffic , 2017, IEEE Transactions on Robotics.

[33]  Alexandre M. Bayen,et al.  Framework for control and deep reinforcement learning in traffic , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[34]  Sergey Levine,et al.  Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations , 2017, Robotics: Science and Systems.

[35]  Razvan Pascanu,et al.  Sim-to-Real Robot Learning from Pixels with Progressive Nets , 2016, CoRL.

[36]  Amnon Shashua,et al.  Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving , 2016, ArXiv.

[37]  Alireza Talebpour,et al.  Influence of connected and autonomous vehicles on traffic flow stability and throughput , 2016 .

[38]  Jian Sun,et al.  Study on traffic characteristics for a typical expressway on-ramp bottleneck considering various merging behaviors , 2015 .

[39]  Zuduo Zheng,et al.  Recent developments and research needs in modeling lane changing , 2014 .

[40]  Jian Sun,et al.  Modeling and Analysis of Merging Behavior at Expressway On-Ramp Bottlenecks , 2014 .

[41]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[42]  Martin Treiber,et al.  Traffic Flow Dynamics: Data, Models and Simulation , 2012 .

[43]  Dirk Helbing,et al.  Adaptive cruise control design for active congestion avoidance , 2008 .

[44]  Chao Wang,et al.  The Effect of Lane-Change Maneuvers on a Simplified Car-Following Theory , 2008, IEEE Transactions on Intelligent Transportation Systems.

[45]  Y. Sugiyama,et al.  Traffic jams without bottlenecks—experimental evidence for the physical mechanism of the formation of a jam , 2008 .

[46]  Haris N. Koutsopoulos,et al.  Integrated driving behavior modeling , 2007 .

[47]  Tomer Toledo,et al.  Modeling Duration of Lane Changes , 2007 .

[48]  Dirk Helbing,et al.  General Lane-Changing Model MOBIL for Car-Following Models , 2007 .

[49]  Peter Hidas,et al.  MODELLING LANE CHANGING AND MERGING IN MICROSCOPIC TRAFFIC SIMULATION , 2002 .

[50]  Helbing,et al.  Congested traffic states in empirical observations and microscopic simulations , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[51]  Doina Precup,et al.  Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning , 1999, Artif. Intell..

[52]  Nakayama,et al.  Dynamical model of traffic congestion and numerical simulation. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[53]  M. Borga Hierarchical Reinforcement Learning , 1993 .

[54]  P. G. Gipps,et al.  A behavioural car-following model for computer simulation , 1981 .

[55]  G. F. Newell Nonlinear Effects in the Dynamics of Car Following , 1961 .

[56]  Sifa Zheng,et al.  Learning-Based Safe Control for Robot and Autonomous Vehicle Using Efficient Safety Certificate , 2023, IEEE Open Journal of Intelligent Transportation Systems.

[57]  J. C. Gerdes,et al.  Learning Policies for Automated Racing Using Vehicle Model Gradients , 2023, IEEE Open Journal of Intelligent Transportation Systems.

[58]  Jian Pu,et al.  Lane Transformer: A High-Efficiency Trajectory Prediction Model , 2023, IEEE Open Journal of Intelligent Transportation Systems.

[59]  Yanggu Zheng,et al.  Comfort-Oriented Motion Planning for Automated Vehicles Using Deep Reinforcement Learning , 2023, IEEE Open Journal of Intelligent Transportation Systems.

[60]  J. Adamy,et al.  Anytime Tree-Based Trajectory Planning for Urban Driving , 2023, IEEE Open Journal of Intelligent Transportation Systems.

[61]  Chin-Teng Lin,et al.  Jointly dampening traffic oscillations and improving energy consumption with electric, connected and automated vehicles: A reinforcement learning based approach , 2020 .

[62]  Nora Sophie Storost,et al.  Tongji University , 2019, Exkursion China 2018.

[63]  Soyoung Ahn,et al.  Traffic dynamics under speed disturbance in mixed traffic with automated and non-automated vehicles , 2019 .

[64]  M. Treiber,et al.  Evidence of convective instability in congested traffic flow: A systematic empirical and theoretical investigation , 2011 .

[65]  Soyoung Ahn,et al.  Freeway traffic oscillations : microscopic analysis of formations and propagations using Wavelet Transform , 2011 .

[66]  Martin Treiber,et al.  Modeling Lane-Changing Decisions with MOBIL , 2009 .

[67]  Moshe Ben-Akiva,et al.  Lane-Changing Model with Explicit Target Lane Choice , 2005 .

[68]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.