D-ACC: Dynamic Adaptive Cruise Control for Highways with On-Ramps Based on Deep Q-Learning

An Adaptive Cruise Control (ACC) system allows vehicles to maintain a desired headway distance to a preceding vehicle automatically. ACC is increasingly adopted by recently available commercial vehicles. Recent research demonstrates that effective use of ACC can improve the traffic flow by adjusting the headway distance in response to dynamically changing traffic conditions. In this paper, we demonstrate that state-of-the-art real-time ACC systems may perform poorly in highway segments with on-ramps because their simple model based only on the traffic conditions of the main road does not take into account the dynamics of merging traffic in determining the optimal headway distance. We propose D-ACC, a dynamic adaptive cruise control system based on deep reinforcement learning that effectively adapts the headway distance according to dynamically changing traffic conditions of both the main road and merging lane to optimize traffic flow. Extensive simulations are performed with a combination of a traffic simulator SUMO and vehicle-to-everything communication (V2X) network simulator Veins under various traffic scenarios. We demonstrate that D-ACC improves the traffic flow by up to 70% compared with a state-of-the-art real-time ACC system in a general highway segment with an on-ramp.

[1]  Zhixia Li,et al.  Fine-tuning ADAS algorithm parameters for optimizing traffic safety and mobility in connected vehicle environment. , 2017, Transportation research. Part C, Emerging technologies.

[2]  Carlo Novara,et al.  Evaluation and Optimization of Adaptive Cruise Control Policies Via Numerical Simulations , 2016, VEHITS.

[3]  Serge P. Hoogendoorn,et al.  A Robust and Efficient Method for Fusing Heterogeneous Data from Traffic Sensors on Freeways , 2010, Comput. Aided Civ. Infrastructure Eng..

[4]  Aurélien Duret,et al.  A hierarchical approach for splitting truck platoons near network discontinuities , 2020, Transportation Research Part B: Methodological.

[5]  L. Davis.  Effect of adaptive cruise control systems on traffic flow. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[6]  Biagio Ciuffo,et al.  Empirical Study on the Properties of Adaptive Cruise Control Systems and Their Impact on Traffic Flow and String Stability , 2020 .

[7]  John M. Dolan,et al.  Learning Highway Ramp Merging Via Reinforcement Learning with Temporally-Extended Actions , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[8]  M. Papageorgiou,et al.  Real time adaptive cruise control strategy for motorways , 2020 .

[9]  Selim Solmaz,et al.  Analysis and Initial Observations on Varying Penetration Rates of Automated Vehicles in Mixed Traffic Flow utilizing SUMO , 2019, 2019 IEEE International Conference on Connected Vehicles and Expo (ICCVE).

[10]  Markos Papageorgiou,et al.  Adaptive Cruise Control Operation for Improved Motorway Traffic Flow , 2018 .

[11]  K. Yuan.  Capacity Drop on Freeways: Traffic Dynamics, Theory and Modeling , 2016 .

[12]  Bart van Arem,et al.  Improving Traffic Flow Efficiency by In-Car Advice on Lane, Speed, and Headway , 2014, IEEE Transactions on Intelligent Transportation Systems.

[13]  Urbano Nunes,et al.  Platooning of autonomous vehicles with intervehicle communications in SUMO traffic simulator , 2010, 13th International IEEE Conference on Intelligent Transportation Systems.

[14]  Jakob Erdmann,et al.  SUMO’s Lane-Changing Model , 2015 .

[15]  Vaneet Aggarwal,et al.  DeepPool: Distributed Model-Free Algorithm for Ride-Sharing Using Deep Reinforcement Learning , 2019, IEEE Transactions on Intelligent Transportation Systems.

[16]  Markos Papageorgiou,et al.  Macroscopic Modelling and Simulation of ACC and CACC Traffic , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[17]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[18]  Francesco Viti,et al.  Impact of different spacing policies for adaptive cruise control on traffic and energy consumption of electric vehicles , 2016, 2016 24th Mediterranean Conference on Control and Automation (MED).

[19]  Markos Papageorgiou,et al.  On Microscopic Modelling of Adaptive Cruise Control Systems , 2015 .

[20]  Sander van Maarseveen.  Impacts of Truck Platooning at Motorway On-ramps: Analysis of traffic performance and safety effects of different platooning strategies and platoon configurations using microscopic simulation , 2017 .

[21]  Michael L. Littman,et al.  Classes of Multiagent Q-learning Dynamics with epsilon-greedy Exploration , 2010, ICML.

[22]  Davis Lc.  Effect of adaptive cruise control systems on traffic flow. , 2004 .

[23]  Xinming Huang,et al.  End-to-end learning for lane keeping of self-driving cars , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[24]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[25]  Ichiro Sakata,et al.  Using advanced adaptive cruise control systems to reduce congestion at sags: An evaluation based on microscopic traffic simulation , 2019, Transportation Research Part C: Emerging Technologies.

[26]  V. K. Katiyar,et al.  Phase transition of traffic states with on-ramp , 2006 .

[27]  András Varga,et al.  An overview of the OMNeT++ simulation environment , 2008, SimuTools.

[28]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[29]  Reinhard German,et al.  Bidirectionally Coupled Network and Road Traffic Simulation for Improved IVC Analysis , 2011, IEEE Transactions on Mobile Computing.

[30]  Azim Eskandarian,et al.  Research advances in intelligent collision avoidance and adaptive cruise control , 2003, IEEE Trans. Intell. Transp. Syst..

[31]  Maxim Raya,et al.  TraCI: an interface for coupling road traffic and network simulators , 2008, CNS '08.

[32]  Serge P. Hoogendoorn,et al.  Empirical Analysis of Merging Behavior at Freeway On-Ramp , 2010 .

[33]  Nikolaos Bekiaris-Liberis,et al.  Feedback Control of Freeway Traffic Flow via Time-Gap Manipulation of ACC-Equipped Vehicles: A PDE-Based Approach , 2019, IFAC-PapersOnLine.

[34]  Markos Papageorgiou,et al.  Simulation of the penetration rate effects of ACC and CACC on macroscopic traffic dynamics , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[35]  Eder Santana,et al.  Learning a Driving Simulator , 2016, ArXiv.