A Deep Reinforcement Learning Method for Pricing Electric Vehicles With Discrete Charging Levels

The effective pricing of electric vehicle (EV) charging by aggregators constitutes a key problem toward the realization of the significant EV flexibility potential in deregulated electricity systems and has been addressed by previous work through bi-level optimization formulations. However, the solution approach adopted in previous work cannot capture the discrete nature of the EV charging/discharging levels. Although reinforcement learning (RL) can tackle this challenge, state-of-the-art RL methods require discretization of state and/or action spaces and thus exhibit limitations in terms of solution optimality and computational requirements. This article proposes a novel deep reinforcement learning (DRL) method to solve the examined EV pricing problem, combining deep deterministic policy gradient (DDPG) principles with a prioritized experience replay (PER) strategy and setting up the problem in multi-dimensional continuous state and action spaces. Case studies demonstrate that the proposed method outperforms state-of-the-art RL methods in terms of both solution optimality and computational requirements and comprehensively analyze the economic impacts of smart-charging and vehicle-to-grid (V2G) flexibility on both aggregators and EV owners.

[1]  Lingwen Gan,et al.  Stochastic distributed protocol for electric vehicle charging with discrete charging rate , 2012, 2012 IEEE Power and Energy Society General Meeting.

[2]  Goran Strbac,et al.  Co-Optimization of Generation Expansion Planning and Electric Vehicles Flexibility , 2016, IEEE Transactions on Smart Grid.

[3]  Jianhui Wang,et al.  Sustainability SI: Optimal Prices of Electricity at Public Charging Stations for Plug-in Electric Vehicles , 2016 .

[4]  Seung Ho Hong,et al.  A Dynamic pricing demand response algorithm for smart grid: Reinforcement learning approach , 2018, Applied Energy.

[5]  Danny H. K. Tsang,et al.  Optimal Scheduling for Electric Vehicle Charging With Discrete Charging Levels in Distribution Grid , 2018, IEEE Transactions on Smart Grid.

[6]  Yelena Vardanyan,et al.  Stochastic Bilevel Program for Optimal Coordinated Energy Trading of an EV Aggregator , 2019, Energies.

[7]  Wolfgang Ketter,et al.  A reinforcement learning approach to autonomous decision-making in smart electricity markets , 2013, Machine Learning.

[8]  José Pablo Chaves-Ávila,et al.  A review of the value of aggregators in electricity systems , 2017 .

[9]  Antonio Liotta,et al.  On-Line Building Energy Optimization Using Deep Reinforcement Learning , 2017, IEEE Transactions on Smart Grid.

[10]  Marko Aunedi,et al.  Cost-Effective Decarbonization in a Decentralized Market: The Benefits of Using Flexible Technologies and Resources , 2019, IEEE Power and Energy Magazine.

[11]  Frank L. Lewis,et al.  Scalable Real-Time Electric Vehicles Charging With Discrete Charging Rates , 2015, IEEE Transactions on Smart Grid.

[12]  Wei Wei,et al.  Energy Pricing and Dispatch for Smart Grid Retailers Under Demand Response and Market Price Uncertainty , 2015, IEEE Transactions on Smart Grid.

[13]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[14]  Danny Pudjianto,et al.  Quantifying the Potential Economic Benefits of Flexible Industrial Demand in the European Power System , 2018, IEEE Transactions on Industrial Informatics.

[15]  Josep M. Guerrero,et al.  A stochastic bi‐level decision‐making framework for a load‐serving entity in day‐ahead and balancing markets , 2019, International Transactions on Electrical Energy Systems.

[16]  Seung Ho Hong,et al.  Incentive-based demand response for smart grid with reinforcement learning and deep neural network , 2019, Applied Energy.

[17]  Josep M. Guerrero,et al.  A Stochastic Bi-Level Scheduling Approach for the Participation of EV Aggregators in Competitive Electricity Markets , 2017 .

[18]  Yang Yu,et al.  Exploring Multi-action Relationship in Reinforcement Learning , 2016, PRICAI.

[19]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[20]  Goran Strbac,et al.  Multi-Period and Multi-Spatial Equilibrium Analysis in Imperfect Electricity Markets: A Novel Multi-Agent Deep Reinforcement Learning Approach , 2019, IEEE Access.

[21]  Sonja Wogrin,et al.  Retail Pricing: A Bilevel Program for PEV Aggregator Decisions Using Indirect Load Control , 2016, IEEE Transactions on Power Systems.

[22]  Filipe Joel Soares,et al.  Integration of Electric Vehicles in the Electric Power System , 2011, Proceedings of the IEEE.

[23]  David B. Richardson,et al.  Electric vehicles and the electric grid: A review of modeling approaches, Impacts, and renewable energy integration , 2013 .

[24]  Willett Kempton,et al.  Vehicle-to-grid power fundamentals: Calculating capacity and net revenue , 2005 .

[25]  Henrik Madsen,et al.  A bilevel model for electricity retailers' participation in a demand response market environment , 2013 .

[26]  Goran Strbac,et al.  Demand side management: Benefits and challenges ☆ , 2008 .

[27]  Haibo He,et al.  Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning , 2019, IEEE Transactions on Smart Grid.

[28]  Peter Vrancx,et al.  Convolutional Neural Networks for Automatic State-Time Feature Extraction in Reinforcement Learning Applied to Residential Load Control , 2016, IEEE Transactions on Smart Grid.

[29]  J. Guerrero,et al.  Optimal Decision-Making Strategy of an Electric Vehicle Aggregator in Short-Term Electricity Markets , 2018, Energies.

[30]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[31]  Tom Schaul,et al.  Prioritized Experience Replay , 2015, ICLR.

[32]  Mihaela van der Schaar,et al.  Dynamic Pricing and Energy Consumption Scheduling With Reinforcement Learning , 2016, IEEE Transactions on Smart Grid.