Indirect Customer-to-Customer Energy Trading With Reinforcement Learning

In this paper, we explore the role of emerging energy brokers (middlemen) in a localized event-driven market (LEM) at the distribution level for facilitating indirect customer-to-customer energy trading. This proposed LEM does not aim to replace any existing energy service or become the best market model; but instead to diversify the energy ecosystem at the edge of distribution networks. In light of this philosophy, the market mechanism will provide additional options for customers and prosumers who have the willingness to directly participate in the retail electricity market occasionally, on top of using existing utility services. It also helps in improving market efficiency and encouraging local-level power balance, while taking into account the characteristics of customers’ behavior. The energy trading process will be built as a Markov decision process with some reinforcement learning and data-driven methods applied. Some economic concepts, like search friction, related to this kind of typical search cost involved market model are also discussed.

[1]  Anzar Mahmood,et al.  Prosumer based energy management and sharing in smart grid , 2018 .

[2]  Cheng Wang,et al.  Energy-Sharing Model With Price-Based Demand Response for Microgrids of Peer-to-Peer Prosumers , 2017, IEEE Transactions on Power Systems.

[3]  Seiji Yasunobu,et al.  Reinforcement learning with nonstationary reward depending on the episode , 2011, 2011 IEEE International Conference on Systems, Man, and Cybernetics.

[4]  W. Marsden.  I and J , 2012 .

[5]  Karen Rudie,et al.  Microgrid level competitive market using dynamic matching , 2013, 2013 IEEE Electrical Power & Energy Conference.

[6]  S. Jacka.  Optimal Stopping and the American Put , 1991 .

[7]  Ann Nowé,et al.  SCANERGY: a Scalable and Modular System for Energy Trading Between Prosumers , 2015, AAMAS.

[8]  B. O. Koopman.  The Theory of Search , 1957 .

[9]  J. Contreras,et al.  Auction Design in Day-Ahead Electricity Markets , 2001, IEEE Power Engineering Review.

[10]  Ildefons Magrans de Abril,et al.  NRGcoin: Virtual currency for trading of renewable energy in smart grids , 2014, International Conference on the European Energy Market.

[11]  Dae-Man Han,et al.  Design and implementation of smart home energy management systems based on zigbee , 2010, IEEE Transactions on Consumer Electronics.

[12]  Chris Watkins,et al.  Learning from delayed rewards , 1989 .

[13]  Chen Yuan,et al.  Co-Optimization Scheme for Distributed Energy Resource Planning in Community Microgrids , 2017, IEEE Transactions on Sustainable Energy.

[14]  Hajir Pourbabak,et al.  The Next-Generation U.S. Retail Electricity Market with Customers and Prosumers—A Bibliographical Survey , 2017 .

[15]  Julia K. Day,et al.  Investigating willingness to save energy and communication about energy use in the American workplace with the attitude-behavior-context model , 2017 .

[16]  J. Neil Bearden,et al.  A new secretary problem with rank-based selection and cardinal payoffs , 2006 .

[17]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[18]  Jianzhong Wu,et al.  Review of Existing Peer-to-Peer Energy Trading Projects , 2017 .

[19]  Thomas Morstyn,et al.  Bilateral Contract Networks for Peer-to-Peer Energy Trading , 2019, IEEE Transactions on Smart Grid.

[20]  Thorsten Joachims,et al.  The K-armed Dueling Bandits Problem , 2012, COLT.

[21]  James Albrecht,et al.  The 2010 Nobel Memorial Prize in Search Theory ∗ , 2011 .

[22]  A. Shwartz,et al.  Handbook of Markov decision processes : methods and applications , 2002 .

[23]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[24]  Daniel M. Johnson,et al.  Gamification and serious games within the domain of domestic energy consumption: A systematic review , 2017 .

[25]  Rupal Rana,et al.  Real-time dynamic pricing in a non-stationary environment using model-free reinforcement learning , 2014 .

[26]  William A. Brock,et al.  Small business economics , 1989 .

[27]  Wencong Su,et al.  The Role of Customers in the U.S. Electricity Market: Past, Present and Future , 2014 .

[28]  John N. Tsitsiklis,et al.  The Complexity of Markov Decision Processes , 1987, Math. Oper. Res..

[29]  Ganguk Hwang,et al.  Event-Driven Energy Trading System in Microgrids: Aperiodic Market Model Analysis With a Game Theoretic Approach , 2017, IEEE Access.