Reinforcement learning for demand response: A review of algorithms and modeling techniques

[1]  José R. Vázquez-Canteli,et al.  Fusing TensorFlow with building energy simulation for intelligent energy management in smart cities , 2019, Sustainable Cities and Society.

[2]  Zoltán Nagy,et al.  LightLearn: An adaptive and occupant centered controller for lighting based on reinforcement learning , 2019, Building and Environment.

[3]  José R. Vázquez-Canteli,et al.  Optimal decarbonization pathways for urban residential building energy services , 2018, Applied Energy.

[4]  Amos J. Storkey,et al.  Augmenting Image Classifiers Using Data Augmentation Generative Adversarial Networks , 2018, ICANN.

[5]  Bo Wang,et al.  Online Markov Chain-based energy management for a hybrid tracked vehicle with speedy Q-learning , 2018, Energy.

[6]  Jianwei Huang,et al.  An Online Learning Algorithm for Demand Response in Smart Grid , 2018, IEEE Transactions on Smart Grid.

[7]  Hongwen He,et al.  Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus , 2018, Applied Energy.

[8]  T. Y. Ji,et al.  Multiple agents and reinforcement learning for modelling charging loads of electric taxis , 2018, Applied Energy.

[9]  Leslie K. Norford,et al.  Optimal control of HVAC and window systems for natural ventilation through reinforcement learning , 2018, Energy and Buildings.

[10]  George A. Vouros,et al.  Fuzzy Q-Learning for multi-agent decentralized energy management in microgrids , 2018, Applied Energy.

[11]  Rui Xiong,et al.  Battery and ultracapacitor in-the-loop approach to validate a real-time power management method for an all-climate electric vehicle , 2018 .

[12]  Jingni Yuan,et al.  Intelligent energy management strategy based on hierarchical approximate global optimization for plug-in fuel cell hybrid electric vehicles , 2018 .

[13]  Victor C. M. Leung,et al.  Mobility-Aware Vehicle-to-Grid Control Algorithm in Microgrids , 2018, IEEE Transactions on Intelligent Transportation Systems.

[14]  Bin Liu,et al.  Fast learning optimiser for real-time optimal energy management of a grid-connected microgrid , 2018 .

[15]  Zoltán Nagy,et al.  Comprehensive analysis of the relationship between thermal comfort and building control research - A data-driven literature review , 2018 .

[16]  Jiayi Cao,et al.  Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle , 2018 .

[17]  Chenming Li,et al.  Energy Management Strategy for a Hybrid Electric Vehicle Based on Deep Reinforcement Learning , 2018 .

[18]  Teng Liu,et al.  A Bi-Level Control for Energy Efficiency Improvement of a Hybrid Tracked Vehicle , 2018, IEEE Transactions on Industrial Informatics.

[19]  Johan Driesen,et al.  Deep Reinforcement Learning based Optimal Control of Hot Water Systems , 2018, ArXiv.

[20]  Peter Vrancx,et al.  Convolutional Neural Networks for Automatic State-Time Feature Extraction in Reinforcement Learning Applied to Residential Load Control , 2016, IEEE Transactions on Smart Grid.

[21]  R. Belmans,et al.  Reinforcement Learning Applied to an Electric Water Heater: From Theory to Practice , 2015, IEEE Transactions on Smart Grid.

[22]  Madeleine Gibescu,et al.  Enabling Cooperative Behavior for Building Demand Response Based on Extended Joint Action Learning , 2018, IEEE Transactions on Industrial Informatics.

[23]  Victor C. M. Leung,et al.  Energy Management in Smart Cities Based on Internet of Things: Peak Demand Reduction and Energy Savings , 2017, Sensors.

[24]  Siobhán Clarke,et al.  Residential demand response: Experimental evaluation and comparison of self-organizing techniques , 2017 .

[25]  Geert Deconinck,et al.  Battery Energy Management in a Microgrid Using Batch Reinforcement Learning , 2017 .

[26]  Chongqing Kang,et al.  Review and prospect of integrated demand response in the multi-energy system , 2017 .

[27]  Ming Zeng,et al.  Impact of behavior-driven demand response on supply adequacy in smart distribution systems , 2017 .

[28]  José R. Vázquez-Canteli,et al.  Balancing comfort and energy consumption of a heat pump using batch reinforcement learning with fitted Q-iteration , 2017 .

[29]  Bart De Schutter,et al.  Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement Learning , 2017, IEEE Transactions on Smart Grid.

[30]  Biao Huang,et al.  A Long-Short Term Memory Recurrent Neural Network Based Reinforcement Learning Controller for Office Heating Ventilation and Air Conditioning Systems , 2017 .

[31]  Tao Yu,et al.  Deep transfer Q-learning with virtual leader-follower for supply-demand Stackelberg game of smart grid , 2017 .

[32]  Mischa Schmidt,et al.  Optimizing legacy building operation: The evolution into data-driven predictive cyber-physical systems , 2017 .

[33]  Yuan Zou,et al.  Implementation of real-time energy management strategy based on reinforcement learning for hybrid electric vehicles and simulation validation , 2017, PloS one.

[34]  Siobhán Clarke,et al.  Prediction-Based Multi-Agent Reinforcement Learning in Inherently Non-Stationary Environments , 2017, ACM Trans. Auton. Adapt. Syst..

[35]  Dongpu Cao,et al.  Reinforcement Learning Optimized Look-Ahead Energy Management of a Parallel Hybrid Electric Vehicle , 2017, IEEE/ASME Transactions on Mechatronics.

[36]  Adriana Chis,et al.  Reinforcement Learning-Based Plug-in Electric Vehicle Charging With Forecasted Price , 2017, IEEE Transactions on Vehicular Technology.

[37]  Jinjun Xiong,et al.  Demand-Side Management of Domestic Electric Water Heaters Using Approximate Dynamic Programming , 2017, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[38]  Derong Liu,et al.  Echo state network-based Q-learning method for optimal battery control of offices combined with renewable energy , 2017 .

[39]  James Brusey,et al.  Reinforcement Learning-based Thermal Comfort Control for Vehicle Cabins , 2017, ArXiv.

[40]  Dirk Vanhoudt,et al.  Model-Free Control of Thermostatically Controlled Loads Connected to a District Heating Network , 2017, ArXiv.

[41]  Pierluigi Siano,et al.  A survey of industrial applications of Demand Response , 2016 .

[42]  Li Xia,et al.  Satisfaction based Q-learning for integrated lighting and blind control , 2016 .

[43]  Muhammad Babar,et al.  Online scheduling of plug-in vehicles in dynamic pricing schemes , 2016 .

[44]  Mihaela van der Schaar,et al.  Dynamic Pricing and Energy Consumption Scheduling With Reinforcement Learning , 2016, IEEE Transactions on Smart Grid.

[45]  Chiara Delmastro,et al.  Generalizable occupant-driven optimization model for domestic hot water production in NZEB , 2016 .

[46]  José R. Vázquez-Canteli,et al.  Massive 3D Models And Physical Data For Building Simulation At The Urban Scale: A Focus On Geneva And Climate Change Scenarios , 2016 .

[47]  Ali Mohammad Ranjbar,et al.  Dynamic load management for a residential customer; Reinforcement Learning approach , 2016 .

[48]  Yuan Zou,et al.  Reinforcement learning-based real-time energy management for a hybrid tracked vehicle , 2016 .

[49]  Mariesa L. Crow,et al.  Heterogeneous Energy Storage Optimization for Microgrids , 2016, IEEE Transactions on Smart Grid.

[50]  Ali Mohammad Ranjbar,et al.  Demand side management for a residential customer in multi-energy systems , 2016 .

[51]  Mohammad Reza Salehizadeh,et al.  Application of fuzzy Q-learning for electricity market modeling by considering renewable power penetration , 2016 .

[52]  Guoyuan Wu,et al.  Data-Driven Reinforcement Learning–Based Real-Time Energy Management System for Plug-In Hybrid Electric Vehicles , 2016 .

[53]  Long Bao Le,et al.  Dynamic Pricing Design for Demand Response Integration in Power Distribution Networks , 2016, IEEE Transactions on Power Systems.

[54]  Stephen Treado,et al.  A review of multi-agent systems concepts and research related to building HVAC control , 2016 .

[55]  Massoud Pedram,et al.  A Near-Optimal Model-Based Control Algorithm for Households Equipped With Residential Photovoltaic Power Generation and Energy Storage Systems , 2016, IEEE Transactions on Sustainable Energy.

[56]  Ding Li,et al.  Machine-Learning Aided Optimal Customer Decisions for an Interactive Smart Grid , 2015, IEEE Systems Journal.

[57]  Luisa F. Cabeza,et al.  Control of a PCM ventilated facade using reinforcement learning techniques , 2015 .

[58]  Guoqiang Zhang,et al.  Control strategies for integration of thermal energy storage into buildings: State-of-the-art review , 2015 .

[59]  Tomohiro Hayashida,et al.  An intelligent Home Energy Management System with classifier system , 2015, 2015 IEEE 8th International Workshop on Computational Intelligence and Applications (IWCIA).

[60]  Lei Yang,et al.  Reinforcement learning for optimal control of low exergy buildings , 2015 .

[61]  Li Xia,et al.  A multi-grid reinforcement learning method for energy conservation and comfort of HVAC in buildings , 2015, 2015 IEEE International Conference on Automation Science and Engineering (CASE).

[62]  Guoyuan Wu,et al.  A Novel Blended Real-Time Energy Management Strategy for Plug-in Hybrid Electric Vehicle Commute Trips , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[63]  Enda Barrett,et al.  Autonomous HVAC Control, A Reinforcement Learning Approach , 2015, ECML/PKDD.

[64]  Peter B. Luh,et al.  Event-Based Optimization Within the Lagrangian Relaxation Framework for Energy Savings in HVAC Systems , 2015, IEEE Transactions on Automation Science and Engineering.

[65]  Shahin Nazarian,et al.  Reinforcement learning-based control of residential energy storage systems for electric bill minimization , 2015, 2015 12th Annual IEEE Consumer Communications and Networking Conference (CCNC).

[66]  Giuseppe Tommaso Costanzo,et al.  Experimental analysis of data-driven control for a building heating system , 2015, ArXiv.

[67]  Naran M. Pindoriya,et al.  Residential Demand Response Algorithms: State-of-the-Art, Key Issues and Challenges , 2015, WISATS.

[68]  Thomas E. Carroll,et al.  Learning based bidding strategy for HVAC systems in double auction retail energy markets , 2015, 2015 American Control Conference (ACC).

[69]  Ali Mohammad Ranjbar,et al.  Applying reinforcement learning method to optimize an Energy Hub operation in the smart grid , 2015, 2015 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT).

[70]  Ronnie Belmans,et al.  Learning Agent for a Heat-Pump Thermostat With a Set-Back Strategy Using Model-Free Reinforcement Learning , 2015, ArXiv.

[71]  Siobhán Clarke,et al.  P-MARL: Prediction-Based Multi-Agent Reinforcement Learning for Non-Stationary Environments , 2015, AAMAS.

[72]  Paras Mandal,et al.  Demand response for sustainable energy systems: A review, application and implementation strategy , 2015 .

[73]  Adriana Chis,et al.  Optimization of plug-in electric vehicle charging with forecasted price , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[74]  Derong Liu,et al.  A Novel Dual Iterative $Q$-Learning Method for Optimal Battery Management in Smart Residential Environments , 2015, IEEE Transactions on Industrial Electronics.

[75]  Chau Yuen,et al.  Electricity Cost Minimization for a Microgrid With Distributed Energy Resource Under Different Information Availability , 2015, IEEE Transactions on Industrial Electronics.

[76]  Tom Holvoet,et al.  Reinforcement Learning of Heuristic EV Fleet Charging in a Day-Ahead Electricity Market , 2015, IEEE Transactions on Smart Grid.

[77]  Muhd Zaimi Abd Majid,et al.  A global review of energy consumption, CO2 emissions and policy in the residential sector (with an overview of the top ten CO2 emitting countries) , 2015 .

[78]  Christos V. Verikoukis,et al.  A Survey on Demand Response Programs in Smart Grids: Pricing Methods and Optimization Algorithms , 2015, IEEE Communications Surveys & Tutorials.

[79]  Anna Helena Reali Costa,et al.  Intelligent Decision-Making for Smart Home Energy Management , 2014, Journal of Intelligent & Robotic Systems.

[80]  Zheng Wen,et al.  Optimal Demand Response Using Device-Based Reinforcement Learning , 2014, IEEE Transactions on Smart Grid.

[81]  R. S. Milton,et al.  Distributed Optimization of Solar Micro-grid Using Multi Agent Reinforcement Learning☆ , 2015 .

[82]  Nadeem Javaid,et al.  A Review on Demand Response: Pricing, Optimization, and Appliance Scheduling , 2015, ANT/SEIT.

[83]  Yunsi Fei,et al.  Smart Home in Smart Microgrid: A Cost-Effective Energy Ecosystem With Intelligent Hierarchical Agents , 2015, IEEE Transactions on Smart Grid.

[84]  Ifiok Otung,et al.  Wireless and Satellite Systems , 2015, Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering.

[85]  Jérôme Henri Kämpf,et al.  A verification of CitySim results using the BESTEST and monitored consumption values , 2015 .

[86]  Jinjun Xiong,et al.  A novel grid load management technique using electric water heaters and Q-learning , 2014, 2014 IEEE International Conference on Smart Grid Communications (SmartGridComm).

[87]  Sudharman K. Jayaweera,et al.  Reinforcement learning aided smart-home decision-making in an interactive smart grid , 2014, 2014 IEEE Green Energy and Systems Conference (IGESC).

[88]  Jin Wen,et al.  Review of building energy modeling for control and operation , 2014 .

[89]  Ronnie Belmans,et al.  Demand response of a heterogeneous cluster of electric water heaters using batch reinforcement learning , 2014, 2014 Power Systems Computation Conference.

[90]  Göran Andersson,et al.  Optimal bidding of plug-in electric vehicles in a market-based control setup , 2014, 2014 Power Systems Computation Conference.

[91]  Mihaela van der Schaar,et al.  Structure-aware stochastic load management in smart grids , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[92]  Siobhán Clarke,et al.  Accelerating Learning in multi-objective systems through Transfer Learning , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[93]  Redouane Lguensat,et al.  Reinforcement Learning Based Algorithm for the Maximization of EV Charging Station Revenue , 2014, 2014 International Conference on Mathematics and Computers in Sciences and in Industry.

[94]  Chang Liu,et al.  Power management for Plug-in Hybrid Electric Vehicles using Reinforcement Learning with trip information , 2014, 2014 IEEE Transportation Electrification Conference and Expo (ITEC).

[95]  R. Belmans,et al.  Impact of residential demand response on power system operation: A Belgian case study , 2014 .

[96]  Ned Djilali,et al.  GridLAB-D: An Agent-Based Simulation Framework for Smart Grids , 2014, J. Appl. Math..

[97]  Ronnie Belmans,et al.  Demand response with locational dynamic pricing to support the integration of renewables , 2014 .

[98]  Kelum A. A. Gamage,et al.  Demand side management in smart grid: A review and proposals for future direction , 2014 .

[99]  Pierluigi Siano,et al.  Demand response and smart grids—A survey , 2014 .

[100]  Derong Liu,et al.  Optimal self-learning battery control in smart residential grids by iterative Q-learning algorithm , 2014, 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL).

[101]  Peggy Seriès,et al.  Reward-Based Learning, Model-Based and Model-Free , 2014, Encyclopedia of Computational Neuroscience.

[102]  Antonio Pietrabissa,et al.  On-board stochastic control of Electric Vehicle recharging , 2013, 52nd IEEE Conference on Decision and Control.

[103]  Christof Weinhardt,et al.  Market-Based EV Charging Coordination , 2013, 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT).

[104]  Peter B. Luh,et al.  Event-based optimization with non-stationary uncertainties to save energy costs of HVAC systems in buildings , 2013, 2013 IEEE International Conference on Automation Science and Engineering (CASE).

[105]  Nandan Parameswaran,et al.  Flexible power consumption management using Q learning techniques in a smart home , 2013, 2013 IEEE Conference on Clean Energy and Technology (CEAT).

[106]  Vinny Cahill,et al.  Multi-agent residential demand response based on load forecasting , 2013, 2013 1st IEEE Conference on Technologies for Sustainability (SusTech).

[107]  Adriana Chis,et al.  Scheduling of plug-in electric vehicle battery charging with price prediction , 2013, IEEE PES ISGT Europe 2013.

[108]  Klaas De Craemer,et al.  Peak shaving of a heterogeneous cluster of residential flexibility carriers using reinforcement learning , 2013, IEEE PES ISGT Europe 2013.

[109]  Long He,et al.  Stochastic Control for Smart Grid Users With Flexible Demand , 2013, IEEE Transactions on Smart Grid.

[110]  Darius Drungilas,et al.  Modelling of Ambient Comfort Affect Reward Based Adaptive Laboratory Climate Controller , 2013 .

[111]  Derong Liu,et al.  Action dependent heuristic dynamic programming for home energy resource scheduling , 2013 .

[112]  William D'haeseleer,et al.  Short-term demand response of flexible electric heating systems: The need for integrated simulations , 2013, 2013 10th International Conference on the European Energy Market (EEM).

[113]  Daniel Urieli,et al.  A learning agent for heat-pump thermostat control , 2013, AAMAS.

[114]  J. Aghaei,et al.  Demand response in smart electricity grids equipped with renewable energy sources: A review , 2013 .

[115]  Sonia Martínez,et al.  Distributed Coverage Games for Energy-Aware Mobile Sensor Networks , 2013, SIAM J. Control. Optim..

[116]  Wolfgang Ketter,et al.  Smart Charging of Electric Vehicles using Reinforcement Learning , 2013, AAAI Workshop: Trading Agent Design and Analysis.

[117]  Marimuthu Palaniswami,et al.  Demand Response Architectures and Load Management Algorithms for Energy-Efficient Power Grids: A Survey , 2012, 2012 Seventh International Conference on Knowledge, Information and Creativity Support Systems.

[118]  Gerhard Weiss,et al.  Multiagent Learning: Basics, Challenges, and Prospects , 2012, AI Mag..

[119]  Jignesh Solanki,et al.  Residential Demand Response model and impact on voltage profile and losses of an electric distribution network , 2012 .

[120]  Warren B. Powell,et al.  An Intelligent Battery Controller Using Bias-Corrected Q-learning , 2012, AAAI.

[121]  José Ramón Gil-García,et al.  Understanding Smart Cities: An Integrative Framework , 2012, HICSS.

[122]  Vincent W. S. Wong,et al.  Real-time vehicle-to-grid control algorithm under price uncertainty , 2011, 2011 IEEE International Conference on Smart Grid Communications (SmartGridComm).

[123]  Yunsi Fei,et al.  Dynamic Residential Demand Response and Distributed Generation Management in Smart Microgrid with Hierarchical Agents , 2011 .

[124]  Heinrich von Stackelberg.  Market Structure and Equilibrium , 2010 .

[125]  Salman Mohagheghi,et al.  Demand Response Architecture: Integration into the Distribution Management System , 2010, 2010 First IEEE International Conference on Smart Grid Communications.

[126]  Marco Levorato,et al.  Residential Demand Response Using Reinforcement Learning , 2010, 2010 First IEEE International Conference on Smart Grid Communications.

[127]  C. Goldman.  Coordination of Energy Efficiency and Demand Response , 2010 .

[128]  Zhen Yu,et al.  Online tuning of a supervisory fuzzy controller for low-energy building system using reinforcement learning , 2010 .

[129]  Minrui Fei,et al.  A two-layer networked learning control system using actor-critic neural network , 2008, Appl. Math. Comput..

[130]  Zhenjun Ma,et al.  Supervisory and Optimal Control of Building HVAC Systems: A Review , 2008 .

[131]  Peter Stone,et al.  Model-Based Reinforcement Learning in a Complex Domain , 2008, RoboCup.

[132]  D. Kolokotsa,et al.  Reinforcement learning for energy conservation and comfort in buildings , 2007 .

[133]  Simeng Liu,et al.  Evaluation of reinforcement learning for optimal control of building active and passive thermal storage inventory , 2007 .

[134]  A. Rosenfeld,et al.  An exploratory analysis of California residential customer response to critical peak pricing of electricity , 2007 .

[135]  Pieter Abbeel,et al.  An Application of Reinforcement Learning to Aerobatic Helicopter Flight , 2006, NIPS.

[136]  Lihong Li,et al.  PAC model-free reinforcement learning , 2006, ICML.

[137]  Simeng Liu,et al.  Experimental analysis of simulated reinforcement learning control for active and passive building thermal storage inventory: Part 2: Results and analysis , 2006 .

[138]  Simeng Liu,et al.  Experimental analysis of simulated reinforcement learning control for active and passive building thermal storage inventory: Part 1. Theoretical foundation , 2006 .

[139]  Johannes Fürnkranz,et al.  Machine Learning: ECML 2006, 17th European Conference on Machine Learning, Berlin, Germany, September 18-22, 2006, Proceedings , 2006, ECML.

[140]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[141]  Dino Pedreschi,et al.  Machine Learning: ECML 2004 , 2004, Lecture Notes in Computer Science.

[142]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[143]  Jing Peng,et al.  Incremental multi-step Q-learning , 1994, Machine Learning.

[144]  Pierre Geurts,et al.  Iteratively Extending Time Horizon Reinforcement Learning , 2003, ECML.

[145]  Gregor P. Henze,et al.  Evaluation of Reinforcement Learning Control for Thermal Energy Storage Systems , 2003 .

[146]  Gregor P. Henze,et al.  Adaptive Optimal Control of a Grid-Independent Photovoltaic System , 2002 .

[147]  Michael C. Mozer,et al.  The Neural Network House: An Environment that Adapts to its Inhabitants , 1998 .

[148]  Douglas C. Hittle,et al.  Synthesis of reinforcement learning, neural networks and PI control applied to a simulated heating coil , 1997, Artificial Intelligence in Engineering.

[149]  Mark Humphrys,et al.  Action Selection methods using Reinforcement Learning , 1996 .

[150]  Leslie Pack Kaelbling,et al.  On the Complexity of Solving Markov Decision Problems , 1995, UAI.

[151]  Stewart W. Wilson.  Classifier Fitness Based on Accuracy , 1995, Evolutionary Computation.

[152]  Michael I. Jordan,et al.  Learning Without State-Estimation in Partially Observable Markovian Decision Processes , 1994, ICML.

[153]  Vijaykumar Gullapalli,et al.  A stochastic reinforcement learning algorithm for learning real-valued functions , 1990, Neural Networks.