Modeling of route planning system based on Q value-based dynamic programming with multi-agent reinforcement learning algorithms

In this paper, a new model for a route planning system based on multi-agent reinforcement learning (MARL) algorithms is proposed. The combined Q-value based dynamic programming (QVDP) with Boltzmann distribution was used to solve vehicle delay's problems by studying the weights of various components in road network environments such as weather, traffic, road safety, and fuel capacity to create a priority route plan for vehicles. The important part of the study was to use a multi-agent system (MAS) with learning abilities which in order to make decisions about routing vehicles between Malaysia's cities. The evaluation was done using a number of case studies that focused on road networks in Malaysia. The results of these experiments indicated that the travel durations for the case studies predicted by existing approaches were between 0.00 and 12.33% off from the actual travel times by the proposed method. From the experiments, the results illustrate that the proposed approach is a unique contribution to the field of computational intelligence in the route planning system.

[1]  Hossein Jula,et al.  Vehicle Route Guidance Systems: Classification and Comparison , 2006, 2006 IEEE Intelligent Transportation Systems Conference.

[2]  Khaled Almejalli,et al.  An intelligent multi-agent approach for road traffic management systems , 2009, 2009 IEEE Control Applications, (CCA) & Intelligent Control, (ISIC).

[3]  Rajarshi Das,et al.  A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation , 2006, 2006 IEEE International Conference on Autonomic Computing.

[4]  Bo Chen,et al.  Integrating mobile agent technology with multi-agent systems for distributed traffic detection and management systems , 2009 .

[5]  Shimon Whiteson,et al.  Multiagent Reinforcement Learning for Urban Traffic Control Using Coordination Graphs , 2008, ECML/PKDD.

[6]  Jian-Min Xu,et al.  A dynamic route guidance arithmetic based on reinforcement learning , 2005, 2005 International Conference on Machine Learning and Cybernetics.

[7]  Abdul Rauf Baig,et al.  Optimization of route planning and exploration using multi agent system , 2010, Multimedia Tools and Applications.

[8]  David Sislák,et al.  LARGE-SCALE HIGH-FIDELITY AGENT-BASED SIMULATION IN AIR TRAFFIC DOMAIN , 2011, Cybern. Syst..

[9]  Bart De Schutter,et al.  Reinforcement Learning and Dynamic Programming Using Function Approximators , 2010 .

[10]  Danny Weyns,et al.  A Decentralized Approach for Anticipatory Vehicle Routing Using Delegate Multiagent Systems , 2011, IEEE Transactions on Intelligent Transportation Systems.

[11]  Makoto Suzuki,et al.  Geographical route planning based on uncertain knowledge , 1995, Proceedings of 7th IEEE International Conference on Tools with Artificial Intelligence.

[12]  Slobodan P. Simonovic,et al.  Optimization of Water Distribution Network Design Using Differential Evolution , 2010 .

[13]  Ruimin Li,et al.  Study on integration of urban traffic control and route guidance based on multi-agent technology , 2003, Proceedings of the 2003 IEEE International Conference on Intelligent Transportation Systems.

[14]  SeowKiam Tian,et al.  Performance of multiagent taxi dispatch on extended-runtime taxi availability , 2010 .

[15]  Daniel Delling,et al.  Engineering and Augmenting Route Planning Algorithms , 2009 .

[16]  Bart De Schutter,et al.  Multi-agent Reinforcement Learning: An Overview , 2010 .

[17]  Sascha Ossowski,et al.  A market-inspired approach to reservation-based urban road traffic management , 2009, AAMAS.

[18]  Yoav Shoham,et al.  Multiagent Systems - Algorithmic, Game-Theoretic, and Logical Foundations , 2009 .

[19]  Ali Selamat,et al.  Multi-agent reinforcement learning for route guidance system , 2011 .

[20]  Horst F. Wedde,et al.  A novel class of multi-agent algorithms for highly dynamic transport planning inspired by honey bee behavior , 2007, 2007 IEEE Conference on Emerging Technologies and Factory Automation (EFTA 2007).

[21]  Stephan Winter,et al.  Towards a Conceptual Model of Talking to a Route Planner , 2008, W2GIS.

[22]  Martin Lauer,et al.  An Algorithm for Distributed Reinforcement Learning in Cooperative Multi-Agent Systems , 2000, ICML.

[23]  Danny Weyns,et al.  Anticipatory Vehicle Routing using Delegate Multi-Agent Systems , 2007, 2007 IEEE Intelligent Transportation Systems Conference.

[24]  Huizhao Tu,et al.  Monitoring Travel Time Reliability on Freeways , 2008 .

[25]  Natalia Akchurina.  Multi-agent reinforcement learning algorithms , 2010 .

[26]  Chen-Khong Tham,et al.  Multi-agent System based Urban Traffic Management , 2007, 2007 IEEE Congress on Evolutionary Computation.

[27]  Michael Wooldridge,et al.  Introduction to multiagent systems , 2001 .

[28]  Zegeye Kebede Gurmu.  A dynamic prediction of travel time for transit time for transit vehicles in Brazil using GPS data , 2010 .

[29]  Nikolay Tchernev,et al.  Urban traffic systems modelling methodology , 2006 .

[30]  Alan Fern,et al.  Bayesian role discovery for multi-agent reinforcement learning , 2010, AAMAS.

[31]  Wu Jigang,et al.  Algorithm for Time-dependent Shortest Safe Path on Transportation Networks , 2011, ICCS.

[32]  Li Jie,et al.  Research on route planning and map-matching in vehicle GPS/dead-reckoning/electronic map integrated navigation system , 2003, Proceedings of the 2003 IEEE International Conference on Intelligent Transportation Systems.

[33]  Wang Heng.  The Route Choice Model Under the Traffic Information Guide Environment Based on Game Theory , 2007 .

[34]  Vikram Manikonda,et al.  A multi-agent approach to cooperative traffic management and route guidance , 2005 .

[35]  Qing-song Li,et al.  Route selecting in the light of the theory of multimode transportation based on multi-agent , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[36]  Alberto Amato,et al.  Multi Agent Negotiation for a Decision Support System in Route Planning , 2008, 2008 International Conference on Computational Intelligence for Modelling Control & Automation.

[37]  Jan D. Gehrke,et al.  Traffic Prediction for Agent Route Planning , 2008, ICCS.

[38]  Dheeraj Kumar,et al.  Multi-Agent System Supply Chain Management in Steel Pipe Manufacturing , 2010 .

[39]  Seyyed Mohsen Hashemi,et al.  Route guidance systems: Review and classification , 2012, 2012 6th Euro American Conference on Telematics and Information Systems (EATIS).

[40]  Yuejin Tan,et al.  Notice of violation of ieee publication principles Dynamic vehicle routing and scheduling with variable travel times in intelligent transportation system , 2006, WCICA 2006.

[41]  Victor J. Blue,et al.  A COOPERATIVE MULTI-AGENT TRANSPORTATION MANAGEMENT AND ROUTE GUIDANCE SYSTEM , 2002 .

[42]  Bart De Schutter,et al.  Multi-agent model predictive control for transportation networks: Serial versus parallel schemes , 2008, Eng. Appl. Artif. Intell..

[43]  Der-Horng Lee,et al.  Towards An Automated Multiagent Taxi-Dispatch System , 2007, 2007 IEEE International Conference on Automation Science and Engineering.

[44]  Valentin Robu,et al.  A multi-agent platform for auction-based allocation of loads in transportation logistics , 2008, Expert Syst. Appl..

[45]  Der-Horng Lee,et al.  Performance of Multiagent Taxi Dispatch on Extended-Runtime Taxi Availability: A Simulation Study , 2010, IEEE Transactions on Intelligent Transportation Systems.

[46]  J. Yu,et al.  Collision-Avoiding Aware Routing Based on Real-Time Hybrid Traffic Infomations , 2011 .

[47]  Chris Tampère,et al.  A MULTI-AGENT CONTROL IN ROAD TRAFFIC MANAGEMENT , 2008 .

[48]  E.H.J. Nijhuis,et al.  Cooperative multi-agent reinforcement learning of traffic lights , 2005 .

[49]  Na Cui,et al.  Simulation and analysis of route guidance strategy based on a multi-agent-game approach , 2008, 2008 International Conference on Management Science and Engineering 15th Annual Conference Proceedings.

[50]  T. Urbanik,et al.  Reinforcement learning-based multi-agent system for network traffic signal control , 2010 .

[51]  M. B. Pellazar.  Vehicle route planning with constraints using genetic algorithms , 1994, Proceedings of National Aerospace and Electronics Conference (NAECON'94).

[52]  Musa Aydin,et al.  Route optimization with Q-learning , 2008 .

[53]  Aura Conci,et al.  A Multi-agent System for Dynamic Path Planning , 2010, 2010 Second Brazilian Workshop on Social Simulation.

[54]  Léon J. M. Rothkrantz,et al.  Dynamic routing using the network of car drivers , 2009, EATIS.

[55]  Vicente R. Tomás López,et al.  A Rule-Based Multi-agent System for Road Traffic Management , 2009, 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology.

[56]  Wilhelm Dangelmaier,et al.  Aspects of Agent Based Planning in the Demand Driven Railcab Scenario , 2007, LDIC.

[57]  Tapas K. Das,et al.  A multi-agent reinforcement learning approach to obtaining dynamic control policies for stochastic lot scheduling problem , 2005, Simul. Model. Pract. Theory.

[58]  Liang Zou,et al.  Application of Genetic Algorithm in Dynamic Route Guidance System , 2007 .

[59]  Lisa Torrey,et al.  Crowd Simulation Via Multi-Agent Reinforcement Learning , 2010, AIIDE.

[60]  Nikos Vlassis,et al.  A Concise Introduction to Multiagent Systems and Distributed Artificial Intelligence I Mobk077-fm Synthesis Lectures on Artificial Intelligence and Machine Learning a Concise Introduction to Multiagent Systems and Distributed Artificial Intelligence a Concise Introduction to Multiagent Systems and D , 2007 .

[61]  Axel Tuma,et al.  Multi-agent-based transport planning in the newspaper industry , 2011 .

[62]  Dariusz Barbucha,et al.  An Agent-Based Guided Local Search for the Capacited Vehicle Routing Problem , 2011, KES-AMSTA.

[63]  Yangsheng Xu,et al.  A simulation study on agent-network based route guidance system , 2005, Proceedings. 2005 IEEE Intelligent Transportation Systems, 2005..

[64]  Mashrur Chowdhury,et al.  Fundamentals of Intelligent Transportation Systems Planning , 2003 .

[65]  Shingo Mabu,et al.  Q value-based Dynamic Programming with SARSA Learning for real time route guidance in large scale road networks , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[66]  Andrew W. Moore,et al.  Reinforcement Learning for Cooperating and Communicating Reactive Agents in Electrical Power Grids , 2000, Balancing Reactivity and Social Deliberation in Multi-Agent Systems.

[67]  Arnošt Motyčka,et al.  Route planning module as a part of Supply Chain Management system , 2012 .

[68]  Toyohide Watanabe,et al.  Vehicle Routing Based on Traffic Cost at Intersection , 2011, KES-AMSTA.

[69]  Bo Chen,et al.  A Review of the Applications of Agent Technology in Traffic and Transportation Systems , 2010, IEEE Transactions on Intelligent Transportation Systems.