Risk-Aware Operation Modeling for Ride-Hailing Fleet in Order Grabbing Mode: A Distributional Reinforcement Learning Approach

With the development of mobility on-demand and transportation electrification technologies, electric vehicle (EV)-based ride-hailing fleets are playing an increasingly important role in the urban ground transportation system. Due to the stochastic nature of order request arrival and electricity price, there exists decision-making risks for ride-hailing EVs operated in order grabbing mode. It is important to investigate their risk-aware operation and model their impact on fleet charging demand and trajectory. In this paper, we propose a distributional reinforcement learning framework to model the risk-aware operation of ride-hailing EVs in order grabbing mode. First, we develop a risk quantification scheme based on the dual theory of choices under risk. Then, we combine Implicit Quantile Network, distorted quantile sampling, and distributional temporal difference learning methods to capture the intrinsic uncertainties and depict the risk-aware EV operation decisions. The proposed framework can provide a more accurate spatial-temporal portrayal of the charging demand and fleet management results. The real-world data from Haikou city is used to illustrate and verify the effectiveness of the proposed scheme.

[1]  Dong-Ki Kang,et al.  Ride-Hailing Service aware Electric Taxi Fleet Management using Reinforcement Learning , 2022, International Conference on Ubiquitous and Future Networks.

[2]  Jieping Ye,et al.  Deep dispatching: A deep reinforcement learning approach for vehicle dispatching on online ride-hailing platform , 2022, Transportation Research Part E: Logistics and Transportation Review.

[3]  Kaishun Wu,et al.  Context-Aware Taxi Dispatching at City-Scale Using Deep Reinforcement Learning , 2022, IEEE Transactions on Intelligent Transportation Systems.

[4]  W. Y. Szeto,et al.  A simulation–optimization framework for a dynamic electric ride-hailing sharing problem with a novel charging strategy , 2022, Transportation Research Part E: Logistics and Transportation Review.

[5]  Jieping Ye,et al.  Multi-Objective Distributional Reinforcement Learning for Large-Scale Order Dispatching , 2021, 2021 IEEE International Conference on Data Mining (ICDM).

[6]  Mohsen Ramezani,et al.  Decentralised cooperative cruising of autonomous ride-sourcing fleets , 2021, Transportation Research Part C: Emerging Technologies.

[7]  Xiaotong Guo,et al.  Robust matching-integrated vehicle rebalancing in ride-hailing system with uncertain demand , 2021, Transportation Research Part B: Methodological.

[8]  Yang Liu,et al.  Optimizing flexible one-to-two matching in ride-hailing systems with boundedly rational users , 2021 .

[9]  H. T. Mouftah,et al.  Intelligent Charging Infrastructure Design for Connected and Autonomous Electric Vehicles in Smart Cities , 2021, 2021 IFIP/IEEE International Symposium on Integrated Network Management (IM).

[10]  Klaus Nökel,et al.  Dynamic planning for simultaneous recharging and relocation of shared electric taxies: A sequential MILP approach , 2021 .

[11]  Aijun Liu,et al.  Optimal operations planning of electric autonomous vehicles via asynchronous learning in ride-hailing systems , 2021 .

[12]  Fan Zhang,et al.  Real-world Ride-hailing Vehicle Repositioning using Deep Reinforcement Learning , 2021, ArXiv.

[13]  Zhaohao Ding,et al.  Mobility-Aware Charging Scheduling for Shared On-Demand Electric Vehicle Fleet Using Deep Reinforcement Learning , 2021, IEEE Transactions on Smart Grid.

[14]  Zeshui Xu,et al.  A novel order evaluation model with nested probabilistic-numerical linguistic information applied to traditional order grabbing mode , 2021, Applied Intelligence.

[15]  Wei Wang,et al.  Operating Electric Vehicle Fleet for Ride-Hailing Services With Reinforcement Learning , 2020, IEEE Transactions on Intelligent Transportation Systems.

[16]  Ramtin Pedarsani,et al.  Dynamic pricing and fleet management for electric autonomous mobility on demand systems , 2020 .

[17]  Fan Zhang,et al.  Ride-Hailing Order Dispatching at DiDi via Reinforcement Learning , 2020, INFORMS J. Appl. Anal..

[18]  Shashi Shekhar,et al.  Risk-aware Energy Management of Extended Range Electric Delivery Vehicles with Implicit Quantile Network , 2020, 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE).

[19]  Yu Xia,et al.  A Mutual Selection Mechanism of Ride-Hailing Based on Hidden Points , 2020, Wirel. Commun. Mob. Comput..

[20]  Hongming Zhu,et al.  Rebalancing Expanding EV Sharing Systems with Deep Reinforcement Learning , 2020, IJCAI.

[21]  Mohammad Shahidehpour,et al.  Deep Reinforcement Learning for EV Charging Navigation by Coordinating Smart Grid and Intelligent Transportation System , 2020, IEEE Transactions on Smart Grid.

[22]  Mohammad Shahidehpour,et al.  Enhanced Coordinated Operations of Electric Power and Transportation Networks via EV Charging Services , 2020, IEEE Transactions on Smart Grid.

[23]  Pradeep Varakantham,et al.  Neural Approximate Dynamic Programming for On-Demand Ride-Pooling , 2019, AAAI.

[24]  Satinder Singh,et al.  Deep Reinforcement Learning for Multi-driver Vehicle Dispatching and Repositioning Problem , 2019, 2019 IEEE International Conference on Data Mining (ICDM).

[25]  Jieping Ye,et al.  Multi-Agent Reinforcement Learning for Order-dispatching via Order-Vehicle Distribution Matching , 2019, CIKM.

[26]  Zhe Xu,et al.  A Deep Value-network Based Approach for Multi-Driver Order Dispatching , 2019, KDD.

[27]  Ning Wang,et al.  Optimizing Order Dispatch for Ride-Sharing Systems , 2019, 2019 28th International Conference on Computer Communication and Networks (ICCCN).

[28]  Jieping Ye,et al.  CoRide: Joint Order Dispatching and Fleet Management for Multi-Scale Ride-Hailing Platforms , 2019, CIKM.

[29]  Marco Pavone,et al.  Scalable and Congestion-aware Routing for Autonomous Mobility-on-Demand via Frank-Wolfe Optimization , 2019, Robotics: Science and Systems.

[30]  Tie-Yan Liu,et al.  A Cooperative Multi-Agent Reinforcement Learning Framework for Resource Balancing in Complex Logistics Network , 2019, AAMAS.

[31]  Hyoshin Park,et al.  A Markov decision process approach to vacant taxi routing with e-hailing , 2019, Transportation Research Part B: Methodological.

[32]  Marco Pavone,et al.  Analysis, Control, and Evaluation of Mobility-on-Demand Systems: A Queueing-Theoretical Approach , 2019, IEEE Transactions on Control of Network Systems.

[33]  Jun Wang,et al.  Efficient Ridesharing Order Dispatching with Mean Field Multi-Agent Reinforcement Learning , 2019, WWW.

[34]  A. Rezaee Jordehi,et al.  How to deal with uncertainties in electric power systems? A review , 2018, Renewable and Sustainable Energy Reviews.

[35]  Marco Pavone,et al.  Stochastic Model Predictive Control for Autonomous Mobility on Demand , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[36]  Zhe Xu,et al.  Efficient Large-Scale Fleet Management via Multi-Agent Deep Reinforcement Learning , 2018, KDD.

[37]  Marc G. Bellemare,et al.  Distributional Reinforcement Learning with Quantile Regression , 2017, AAAI.

[38]  M. Pavone,et al.  Data-Driven Model Predictive Control of Autonomous Mobility-on-Demand Systems , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[39]  Marc G. Bellemare,et al.  The Cramer Distance as a Solution to Biased Wasserstein Gradients , 2017, ArXiv.

[40]  Marco Pavone,et al.  A BCMP network approach to modeling and controlling autonomous mobility-on-demand systems , 2016, Int. J. Robotics Res..

[41]  Marco Pavone,et al.  Routing autonomous vehicles in congested transportation networks: structural properties and coordination algorithms , 2016, Autonomous Robots.

[42]  Emilio Frazzoli,et al.  Models, algorithms, and evaluation for autonomous mobility-on-demand systems , 2015, 2015 American Control Conference (ACC).

[43]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[44]  Marco Pavone,et al.  Control of robotic mobility-on-demand systems: A queueing-theoretical perspective , 2014, Int. J. Robotics Res..

[45]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[46]  Shaun S. Wang.  A CLASS OF DISTORTION OPERATORS FOR PRICING FINANCIAL AND INSURANCE RISKS , 2000 .

[47]  Doina Precup,et al.  Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning , 1999, Artif. Intell..

[48]  Andreas A. Malikopoulos,et al.  Congestion-Aware Routing, Rebalancing, and Charging Scheduling for Autonomous Electric Mobility-on-Demand System , 2021 .

[49]  Rainer Schlosser,et al.  Predicting Location Probabilities of Drivers to Improve Dispatch Decisions of Transportation Network Companies based on Trajectory Data , 2020, ICORES.

[50]  杨 家隆,et al.  Research on Service of E Vehicle Platform Based on Order Grabbing Mode and Omni-Directional Performance Management System , 2019 .

[51]  Mary R. Hardy,et al.  Distortion Risk Measures : Coherence and Stochastic Dominance , 2002 .

[52]  G. Pflug.  Some Remarks on the Value-at-Risk and the Conditional Value-at-Risk , 2000 .

[53]  M. Yaari.  The Dual Theory of Choice under Risk , 1987 .

[54]  S. Sen,et al.  Stochastic Programming: Computational Issues and Challenges , 2022 .