DHP Method for Ramp Metering of Freeway Traffic

This paper presents the design of dual heuristic programming (DHP) for the optimal coordination of ramp metering in freeway systems. Specifically, we implement the DHP method to solve both recurrent and nonrecurrent congestions with queuing consideration. A coordinated neural network controller is achieved by the DHP method with traffic models. Then, it is used for verifications with different traffic scenarios. Simulation studies performed on a hypothetical freeway indicate that the achieved neural controller maintains good control performance when compared with the classical ramp metering algorithm ALINEA. We emphasize that these neural controllers can be developed offline by using approximate traffic models. This offline mechanism avoids the risks of instability that incur during continual online training. We also discuss some real-time implementation issues.

[1]  T. Shannon,et al.  Qualitative models for adaptive critic neurocontrol , 1999, IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028).

[2]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[3]  Markos Papageorgiou,et al.  Modelling and real-time control of traffic flow on the southern part of Boulevard Peripherique in Paris: Part II: Coordinated on-ramp metering , 1990 .

[4]  Markos Papageorgiou,et al.  Applications of Automatic Control Concepts to Traffic Flow Modeling and Control , 1983 .

[5]  Donald C. Wunsch,et al.  Convergence of critic-based training , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[6]  R. Bellman.  Dynamic programming. , 1957, Science.

[7]  G. Saridis,et al.  Suboptimal control for nonlinear stochastic systems , 1992, [1992] Proceedings of the 31st IEEE Conference on Decision and Control.

[8]  Tom Bellemans,et al.  Model predictive control for ramp metering of motorway traffic: A case study , 2006 .

[9]  Stephen G. Ritchie,et al.  Coordinated traffic-responsive ramp control via nonlinear state feedback , 2001 .

[10]  N. B. Goldstein,et al.  A decentralized control strategy for freeway regulation , 1982 .

[11]  Wang,et al.  Review of road traffic control strategies , 2003, Proceedings of the IEEE.

[12]  Markos Papageorgiou,et al.  COORDINATED AND INTEGRATED CONTROL OF MOTORWAY NETWORKS VIA NON-LINEAR OPTIMAL CONTROL , 2002 .

[13]  Paul J. Werbos,et al.  Approximate dynamic programming for real-time control and neural modeling , 1992 .

[14]  Jennie Si,et al.  Online learning control by association and reinforcement , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[15]  Danil V. Prokhorov,et al.  Analyzing for Lyapunov stability with adaptive critics , 1998, SMC'98 Conference Proceedings. 1998 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.98CH36218).

[16]  Jing Xu,et al.  Coordinated Control of Multiple Ramp Metering Based on DHP (λ) Controller , 2008, 2008 11th International IEEE Conference on Intelligent Transportation Systems.

[17]  Derong Liu,et al.  Action-dependent adaptive critic designs , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[18]  Derong Liu,et al.  Direct Neural Dynamic Programming , 2004 .

[19]  Kenji Doya,et al.  Reinforcement Learning in Continuous Time and Space , 2000, Neural Computation.

[20]  James Robinson,et al.  RAMP METERING STATUS IN NORTH AMERICA. , 1989 .

[21]  G. Saridis,et al.  On Successive Approximation of Optimal Control of Stochastic Dynamic Systems , 2005 .

[22]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[23]  Donald A. Sofge,et al.  Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches , 1992 .

[24]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[25]  Markos Papageorgiou,et al.  ALINEA: A LOCAL FEEDBACK CONTROL LAW FOR ON-RAMP METERING , 1990 .

[26]  M Cremer,et al.  AN EXTENDED TRAFFIC MODEL FOR FREEWAY CONTROL , 1985 .

[27]  Chin-Teng Lin,et al.  Reinforcement learning for an ART-based fuzzy adaptive learning control network , 1996, IEEE Trans. Neural Networks.

[28]  Markos Papageorgiou,et al.  Multilayer control system design applied to freeway traffic , 1984 .

[29]  Jianqiang Yi,et al.  Coordinated multiple ramps metering based on neuro-fuzzy adaptive dynamic programming , 2009, 2009 International Joint Conference on Neural Networks.

[30]  Thomas Parisini,et al.  Neural approximations for feedback optimal control of freeway systems , 2001, IEEE Trans. Veh. Technol..

[31]  George G. Lendaris,et al.  Adaptive dynamic programming , 2002, IEEE Trans. Syst. Man Cybern. Part C.

[32]  George G. Lendaris,et al.  Training strategies for critic and action neural networks in dual heuristic programming method , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[33]  Bart De Schutter,et al.  IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS Editor-In-Chief , 2005 .

[34]  Bart De Schutter,et al.  Model predictive control for optimal coordination of ramp metering and variable speed limits , 2005 .

[35]  Yi-Hsien Chiang,et al.  Control of Freeway Traffic Flow in Unstable Phase by $H_{\infty}$ Theory , 2008, IEEE Transactions on Intelligent Transportation Systems.

[36]  Markos Papageorgiou,et al.  Freeway ramp metering: an overview , 2002, IEEE Trans. Intell. Transp. Syst..

[37]  Harold J Payne,et al.  MODELS OF FREEWAY TRAFFIC AND CONTROL. , 1971 .

[38]  Baher Abdulhai,et al.  Automated Adaptive Traffic Corridor Control Using Reinforcement Learning: Approach and Case Studies , 2006 .

[39]  Markos Papageorgiou,et al.  ALINEA Local Ramp Metering: Summary of Field Results , 1997 .

[40]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[41]  Harold Chestnut.  The International Federation of Automatic Control , 1960 .

[42]  Perry Y. Li,et al.  Traffic flow stabilization , 1995, Proceedings of 1995 American Control Conference - ACC'95.