Big data driven smart energy management: From big data to big insights

Large amounts of data are increasingly accumulated in the energy sector with the continuous application of sensors, wireless transmission, network communication, and cloud computing technologies. To fulfill the potential of energy big data and obtain insights to achieve smart energy management, we present a comprehensive study of big data driven smart energy management. We first discuss the sources and characteristics of energy big data. Also, a process model of big data driven smart energy management is proposed. Then taking smart grid as the research background, we provide a systematic review of big data analytics for smart energy management. It is discussed from four major aspects, namely power generation side management, microgrid and renewable energy management, asset management and collaborative operation, as well as demand side management (DSM). Afterwards, the industrial development of big data-driven smart energy management is analyzed and discussed. Finally, we point out the challenges of big data-driven smart energy management in IT infrastructure, data collection and governance, data integration and sharing, processing and analysis, security and privacy, and professionals.

[1]  Shanlin Yang,et al.  Energy conservation and emission reduction of China’s electric power industry , 2015 .

[2]  M. Shahidehpour,et al.  Time management for assets: chronological strategies for power system asset management , 2005, IEEE Power and Energy Magazine.

[3]  Young-Jin Kim,et al.  A secure decentralized data-centric information infrastructure for smart grid , 2010, IEEE Communications Magazine.

[4]  Ning Lu,et al.  Smart-grid security issues , 2010, IEEE Security & Privacy.

[5]  Sakti Prasad Ghoshal,et al.  Solution of combined economic and emission dispatch problems of power systems by an opposition-based harmony search algorithm , 2012 .

[6]  V. Kachitvichyanukul,et al.  A New Efficient GA-Benders' Decomposition Method: For Power Generation Expansion Planning With Emission Controls , 2007, IEEE Transactions on Power Systems.

[7]  Timothy C. Green,et al.  Intermittent renewable generation and the cost of maintaining power system reliability , 2008 .

[8]  Nei Kato,et al.  Toward intelligent machine-to-machine communications in smart grid , 2011, IEEE Communications Magazine.

[9]  C. Reinhart,et al.  A method for predicting city-wide electricity gains from photovoltaic panels based on LiDAR and GIS data combined with hourly Daysim simulations , 2013 .

[10]  S. Borenstein.  The Long-Run Efficiency of Real-Time Electricity Pricing , 2005 .

[11]  C. Senabre,et al.  Classification, Filtering, and Identification of Electrical Customer Load Patterns Through the Use of Self-Organizing Maps , 2006, IEEE Transactions on Power Systems.

[12]  Manfred Morari,et al.  Reducing peak electricity demand in building climate control using real-time pricing and model predictive control , 2010, 49th IEEE Conference on Decision and Control (CDC).

[13]  Chung-Hsiao Wang,et al.  Electric power generation planning for interrelated projects: a real options approach , 2006, IEEE Transactions on Engineering Management.

[14]  Mladen Kezunovic,et al.  Automated monitoring functions for improved power system operation and control , 2005 .

[15]  S. Awerbuch.  Portfolio-Based Electricity Generation Planning: Policy Implications For Renewables And Energy Security , 2006 .

[16]  Roy Billinton,et al.  Sequential Monte Carlo simulation for composite power system reliability analysis with time varying loads , 1995 .

[17]  Gianfranco Chicco,et al.  Overview and performance assessment of the clustering methods for electrical load pattern grouping , 2012 .

[18]  H.N. Koivo,et al.  System Modelling and Online Optimal Management of MicroGrid Using Multiobjective Optimization , 2007, 2007 International Conference on Clean Electrical Power.

[19]  Shanlin Yang,et al.  Comparison and Weighted Summation Type of Fuzzy Cluster Validity Indices , 2014, Int. J. Comput. Commun. Control.

[20]  Viktor Mayer-Schnberger,et al.  Big Data: A Revolution That Will Transform How We Live, Work, and Think , 2013 .

[21]  Hossein Shayeghi,et al.  Iteration particle swarm optimization procedure for economic load dispatch with generator constraints , 2011, Expert Syst. Appl..

[22]  Borut Mavko,et al.  Application of the fault tree analysis for assessment of power system reliability , 2009, Reliab. Eng. Syst. Saf..

[23]  Farshid Keynia,et al.  A New Neural Network Approach to Short Term Load Forecasting of Electrical Power Systems , 2011 .

[24]  Elisabeth Dütschke,et al.  Dynamic electricity pricing - Which programs do consumers prefer? , 2013 .

[25]  Hirofumi Matsuo,et al.  Standalone Hybrid Wind-Solar Power Generation System Applying Dump Power Control Without Dump Load , 2012, IEEE Transactions on Industrial Electronics.

[26]  Aoife Foley,et al.  Current methods and advances in forecasting of wind power generation , 2012 .

[27]  Yuting Wang,et al.  Very Short-Term Load Forecasting: Wavelet Neural Networks With Data Pre-Filtering , 2013, IEEE Transactions on Power Systems.

[28]  Sheng-wei Fei,et al.  Fault diagnosis of power transformer based on support vector machine with genetic algorithm , 2009, Expert Syst. Appl..

[29]  Paul Zikopoulos,et al.  Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data , 2011 .

[30]  Tomonobu Senjyu,et al.  Solving economic load dispatch problem with valve-point effects using a hybrid quantum mechanics inspired particle swarm optimisation , 2011 .

[31]  Bijaya Ketan Panigrahi,et al.  Adaptive particle swarm optimization approach for static and dynamic economic load dispatch , 2008 .

[32]  Zhang Zhisheng.  Short Communication: Quantum-behaved particle swarm optimization algorithm for economic load dispatch of power system , 2010 .

[33]  Wang Fan,et al.  An approach to power transformer asset management using health index , 2009, IEEE Electrical Insulation Magazine.

[34]  Fan Yang,et al.  A power efficient 1.0625-3.125 Gb/s serial transceiver in 130 nm digital CMOS for multi-standard applications , 2013, Science China Information Sciences.

[35]  Henrik Madsen,et al.  Using quantile regression to extend an existing wind power forecasting system with probabilistic forecasts , 2006 .

[36]  H.N. Koivo,et al.  MicroGrid Online Management and Balancing Using Multiobjective Optimization , 2007, 2007 IEEE Lausanne Power Tech.

[37]  Seddik Bacha,et al.  Forecasting photovoltaic array power production subject to mismatch losses , 2010 .

[38]  S.K. Tiong,et al.  Non-Technical Loss analysis for detection of electricity theft using support vector machines , 2008, 2008 IEEE 2nd International Power and Energy Conference.

[39]  Wenyuan Li,et al.  Power System Operation Risk Assessment Using Credibility Theory , 2008, IEEE Transactions on Power Systems.

[40]  Stefano Spaccapietra,et al.  Issues and approaches of database integration , 1998, CACM.

[41]  Patrick D. McDaniel,et al.  Security and Privacy Challenges in the Smart Grid , 2009, IEEE Security & Privacy.

[42]  João Paulo Papa,et al.  A novel algorithm for feature selection using Harmony Search and its application for non-technical losses detection , 2011, Comput. Electr. Eng..

[43]  Heikki N. Koivo,et al.  Online Management of MicroGrid with Battery Storage Using Multiobjective Optimization , 2007, 2007 International Conference on Power Engineering, Energy and Electrical Drives.

[44]  Andreas Ehrenmann,et al.  Space and Time: Wind in an Investment Planning Model , 2006 .

[45]  Robert J. Marks,et al.  Electric load forecasting using an artificial neural network , 1991 .

[46]  Wei-Chiang Hong,et al.  Electric load forecasting by seasonal recurrent SVR (support vector regression) with chaotic artific , 2011 .

[47]  Z. Zakaria,et al.  Two-stage fuzzy clustering approach for load profiling , 2009, 2009 44th International Universities Power Engineering Conference (UPEC).

[48]  H. Pedro,et al.  Assessment of forecasting techniques for solar power production with no exogenous inputs , 2012 .

[49]  M. Parsa Moghaddam,et al.  Modeling and prioritizing demand response programs in power markets , 2010 .

[50]  Magdy M. A. Salama,et al.  Asset management techniques for transformers , 2010 .

[51]  Xisheng Tang,et al.  Research of Power Generator Planning Based on Low-Carbon Targets , 2011, 2011 Third Pacific-Asia Conference on Circuits, Communications and System (PACCS).

[52]  P. K. Chattopadhyay,et al.  Solving complex economic load dispatch problems using biogeography-based optimization , 2010, Expert Syst. Appl..

[53]  Zhou Kai-le.  An improved fuzzy C-means algorithm for power load characteristics classification , 2012 .

[54]  Mikko Kolehmainen,et al.  Data-based method for creating electricity use load profiles using large amount of customer-specific hourly measured electricity use data , 2010 .

[55]  M. R. AlRashidi,et al.  LONG TERM ELECTRIC LOAD FORECASTING BASED ON PARTICLE SWARM OPTIMIZATION , 2010 .

[56]  C. Bennett,et al.  Networking AMI Smart Meters , 2008, 2008 IEEE Energy 2030 Conference.

[57]  Vishnu Prasad,et al.  Economic dispatch using particle swarm optimization: A review , 2009 .

[58]  S. Fan,et al.  Short-term load forecasting based on an adaptive hybrid method , 2006, IEEE Transactions on Power Systems.

[59]  Khosrow Moslehi,et al.  A Reliability Perspective of the Smart Grid , 2010, IEEE Transactions on Smart Grid.

[60]  Zhang Yan,et al.  A review on the forecasting of wind speed and generated power , 2009 .

[61]  Carlos Arthur Mattos Teixeira Cavalcante,et al.  A new method for pattern recognition in load profiles to support decision-making in the management of the electric sector , 2013 .

[62]  E. Muela,et al.  Fuzzy possibilistic model for medium-term power generation planning with environmental criteria , 2007 .

[63]  P. K. Chattopadhyay,et al.  Biogeography-Based Optimization for Different Economic Load Dispatch Problems , 2010, IEEE Transactions on Power Systems.

[64]  Wei Sun,et al.  Application of neural network model combining information entropy and ant colony clustering theory for short-term load forecasting , 2005, 2005 International Conference on Machine Learning and Cybernetics.

[65]  James A. Momoh,et al.  Smart grid design for efficient and flexible power networks operation and control , 2009, 2009 IEEE/PES Power Systems Conference and Exposition.

[66]  Hung-po Chao,et al.  Efficient pricing and investment in electricity markets with intermittent resources , 2011 .

[67]  Haiyan Lu,et al.  Combined modeling for electric load forecasting with adaptive particle swarm optimization , 2010 .

[68]  M. Amin,et al.  Challenges in reliability, security, efficiency, and resilience of energy infrastructure: Toward smart self-healing electric power grid , 2008, 2008 IEEE Power and Energy Society General Meeting - Conversion and Delivery of Electrical Energy in the 21st Century.

[69]  F. Muñoz,et al.  Hopfield K-Means clustering algorithm: A proposal for the segmentation of electricity customers , 2011 .

[70]  Yannig Goude,et al.  Modeling and Forecasting Daily Electricity Load Curves: A Hybrid Approach , 2013, 1611.08632.

[71]  W. El-khattam,et al.  Optimal investment planning for distributed generation in a competitive electricity market , 2004, IEEE Transactions on Power Systems.

[72]  Wei-Chiang Hong,et al.  Application of chaotic ant swarm optimization in electric load forecasting , 2010 .

[73]  Ramesh C. Bansal,et al.  Economic Load Dispatch Using Hybrid Swarm Intelligence Based Harmony Search Algorithm , 2011 .

[74]  A.S. Bretas,et al.  Hybrid Fault Diagnosis Scheme Implementation for Power Distribution Systems Automation , 2008, IEEE Transactions on Power Delivery.

[75]  Henrik Madsen,et al.  Online short-term solar power forecasting , 2009 .

[76]  N.D. Hatziargyriou,et al.  An Advanced Statistical Method for Wind Power Forecasting , 2007, IEEE Transactions on Power Systems.

[77]  Iñigo Monedero,et al.  Detection of frauds and other non-technical losses in a power utility using Pearson coefficient, Bayesian networks and decision trees , 2012 .

[78]  B.F. Wollenberg,et al.  Toward a smart grid: power delivery for the 21st century , 2005, IEEE Power and Energy Magazine.

[79]  Yaoyu Li,et al.  A review of recent advances in wind turbine condition monitoring and fault diagnosis , 2009, 2009 IEEE Power Electronics and Machines in Wind Applications.

[80]  N. Amjady,et al.  Short-Term Bus Load Forecasting of Power Systems by a New Hybrid Method , 2007, IEEE Transactions on Power Systems.

[81]  Qixin Chen,et al.  Power Generation Expansion Planning Model Towards Low-Carbon Economy and Its Application in China , 2010, IEEE Transactions on Power Systems.

[82]  A. G. Kagiannas,et al.  Power generation planning: a survey from monopoly to competition , 2004 .

[83]  A. Kusiak,et al.  Short-Term Prediction of Wind Farm Power: A Data Mining Approach , 2009, IEEE Transactions on Energy Conversion.

[84]  Ying Chen,et al.  Short-Term Load Forecasting: Similar Day-Based Wavelet Neural Networks , 2010, IEEE Transactions on Power Systems.

[85]  Bangyin Liu,et al.  Online 24-h solar power forecasting based on weather type classification using artificial neural network , 2011 .

[86]  Martin Greiner,et al.  Seasonal optimal mix of wind and solar power in a future, highly renewable Europe , 2010 .

[87]  Silja Meyer-Nieberg,et al.  Electric load forecasting methods: Tools for decision making , 2009, Eur. J. Oper. Res..

[88]  H. Farhangi,et al.  The path of the smart grid , 2010, IEEE Power and Energy Magazine.

[89]  L. Bertling,et al.  A reliability-centered asset maintenance method for assessing the impact of maintenance in power distribution systems , 2005, IEEE Transactions on Power Systems.

[90]  A. Faruqui,et al.  Household response to dynamic pricing of electricity: a survey of 15 experiments , 2010 .

[91]  Dionysis Assimacopoulos,et al.  Evaluation of Renewable Energy potential using a GIS decision support system , 1998 .

[92]  H. M. I. Pousinho,et al.  An Artificial Neural Network Approach for Short-Term Wind Power Forecasting in Portugal , 2009, 2009 15th International Conference on Intelligent System Applications to Power Systems.

[93]  Desheng Dash Wu,et al.  Power load forecasting using support vector machine and ant colony optimization , 2010, Expert Syst. Appl..

[94]  Vincent W. S. Wong,et al.  Optimal Real-Time Pricing Algorithm Based on Utility Maximization for Smart Grid , 2010, 2010 First IEEE International Conference on Smart Grid Communications.

[95]  Rajesh Kumar Nema,et al.  A current and future state of art development of hybrid energy system using wind and PV-solar: A review , 2009 .

[96]  A. Gomes Martins,et al.  A multiple objective mixed integer linear programming model for power generation expansion planning , 2004 .

[97]  F. Gubina,et al.  Determination and allocation of typical load profiles to the eligible consumers , 2003, 2003 IEEE Bologna Power Tech Conference Proceedings,.

[98]  Chao Shen,et al.  A review of electric load classification in smart grid environment , 2013 .

[99]  R. Teodorescu,et al.  Overview of recent grid codes for wind power integration , 2010, 2010 12th International Conference on Optimization of Electrical and Electronic Equipment.

[100]  Vahid Vahidinasab,et al.  A modified harmony search method for environmental/economic load dispatch of real-world power systems , 2014 .

[101]  H. Iba,et al.  Differential evolution for economic load dispatch problems , 2008 .

[102]  P. K. Chattopadhyay,et al.  Hybrid Differential Evolution With Biogeography-Based Optimization for Solution of Economic Load Dispatch , 2010, IEEE Transactions on Power Systems.

[103]  Shanlin Yang,et al.  A framework of service-oriented operation model of China׳s power system , 2015 .

[104]  Kara Clark,et al.  How do Wind and Solar Power Affect Grid Operations: The Western Wind and Solar Integration Study; Preprint , 2009 .

[105]  Shanlin Yang,et al.  Optimal load distribution model of microgrid in the smart grid environment , 2014 .

[106]  Fábio Gonçalves Jota,et al.  Building load management using cluster and statistical analyses , 2011 .

[107]  Hamed Mohsenian Rad,et al.  Optimal Residential Load Control With Price Prediction in Real-Time Electricity Pricing Environments , 2010, IEEE Transactions on Smart Grid.

[108]  R.H. Lasseter,et al.  Microgrid: a conceptual solution , 2004, 2004 IEEE 35th Annual Power Electronics Specialists Conference (IEEE Cat. No.04CH37551).

[109]  Vasant Honavar,et al.  A Service-Oriented Architecture for Electric Power Transmission System Asset Management , 2006, ICSOC Workshops.

[110]  Stefano Spaccapietra,et al.  On Spatial Database Integration , 1998, Int. J. Geogr. Inf. Sci..

[111]  Ling Wang,et al.  An effective differential harmony search algorithm for the solving non-convex economic load dispatch problems , 2013 .

[112]  Shanlin Yang,et al.  Fuzziness parameter selection in fuzzy c-means: The perspective of cluster validation , 2014, Science China Information Sciences.

[113]  Ken Nagasaka,et al.  Neural Network Ensemble-Based Solar Power Generation Short-Term Forecasting , 2009, J. Adv. Comput. Intell. Intell. Informatics.

[114]  A.H. Nizar,et al.  Load Profiling Method in Detecting non-Technical Loss Activities in a Power Utility , 2006, 2006 IEEE International Power and Energy Conference.

[115]  A. Grandjean,et al.  A review and an analysis of the residential electric load curve models , 2012 .

[116]  Shanlin Yang,et al.  Demand side management in China: The context of China’s power industry reform , 2015 .

[117]  Rob J Hyndman,et al.  Short-Term Load Forecasting Based on a Semi-Parametric Additive Model , 2012, IEEE Transactions on Power Systems.

[118]  Gerard J. M. Smit,et al.  Management and Control of Domestic Smart Grid Technology , 2010, IEEE Transactions on Smart Grid.

[119]  Dejing Dou,et al.  Knowledge Representation Formalisms and Methods—Representation , 2022 .