Integrated optimal control strategies for freeway traffic mixed with connected automated vehicles: A model-based reinforcement learning approach

Abstract This paper proposes an integrated freeway traffic flow control framework that aims to minimize the total travel cost, improve greenness and safety for freeway traffic mixed with connected automated vehicles (CAVs) and regular human-piloted vehicles (RHVs). The proposed framework devises an integrated action of several control strategies such as ramp metering, lane changing control (LCC) for CAVs and lane changing recommendation (LCR) for RHVs, variable speed limit control (VSLC) for CAVs and variable speed limit recommendation (VSLR) for RHVs with minimum safety gap control measures for lane changing and merging maneuvers. The CAVs are assumed to follow the system control instructions fully and immediately. In contrast, the RHVs would make decisions in response to the recommendations disseminated and also the behaviors of CAVs. The compliance rate of drivers to the LCR is captured by the underlying traffic flow model. A set of constraints is imposed to restrict VSLC/VSLR and LCC/LCR measures from changing too frequently or too sharply on both temporal and spatial dimensions to avoid excessive nuisance to passengers and traffic flow instability. A reinforcement learning based solution algorithm is proposed. First, a control parameterization technique is adopted to reduce the dimension of the original optimal control problem to increase computational efficiency. Then, a gradient-free Cross-Entropy-Method based algorithm is used to search the optimal parameters to circumvent the non-differentiability of the traffic flow model. The feasibility and effectiveness of the proposed framework are illustrated via numerical examples for a variety of penetration rates of CAVs under various traffic conditions. A sensitivity analysis is conducted to demonstrate the impacts of several important parameters such as the reaction time of the CAVs. It is found that the integrated control strategy can reduce the total travel cost by reducing the lane changing maneuvers and vehicles queuing at the bottleneck meanwhile smooth the traffic flow and suppress the adverse impact of shockwaves. The effect of ramp metering is not significant when the penetration rate of CAVs is high enough. Speed harmonization (with minimum gap control) in conjunction with LCC/LCR would be a better integrated control strategy under high penetration rate of CAVs.