Big data framework for analytics in smart grids

Abstract Smart meters are being deployed replacing conventional meters worldwide and to enable automated collection of energy consumption data. However, the massive amounts of data evolving from smart grid meters used for monitoring and control purposes need to be sufficiently managed to increase the efficiency, reliability and sustainability of the smart grid. Interestingly, the nature of smart grids can be considered as a big data challenge that requires advanced informatics techniques and cyber-infrastructure to deal with huge amounts of data and their analytics. For that, this unprecedented smart grid data require an effective platform that takes the smart grid a step forward in the big data era. This paper presents a framework that can be a start for innovative research and take smart grids a step forward. An implementation of the framework on a secure cloud-based platform is presented. Furthermore, the framework has been applied on two scenarios to visualize the energy, for a single-house and a smart grid that contains over 6000 smart meters. The application of the two scenarios to visualize the grid status and enable dynamic demand response, suggests that the framework is feasible in performing further smart grid data analytics.

[1]  Josune Hernantes,et al.  Infrastructure as a Service and Cloud Technologies , 2015, IEEE Software.

[2]  Martin Grund,et al.  Impala: A Modern, Open-Source SQL Engine for Hadoop , 2015, CIDR.

[3]  Cees T. A. M. de Laat,et al.  Defining architecture components of the Big Data Ecosystem , 2014, 2014 International Conference on Collaboration Technologies and Systems (CTS).

[4]  Hortensia Amaris,et al.  Optimal Charging Scheduling of Electric Vehicles in Smart Grids by Heuristic Algorithms , 2014 .

[5]  Avita Katal,et al.  Big data: Issues, challenges, tools and Good practices , 2013, 2013 Sixth International Conference on Contemporary Computing (IC3).

[6]  GhemawatSanjay,et al.  The Google file system , 2003 .

[7]  Paul Zikopoulos,et al.  Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data , 2011 .

[8]  Tom White,et al.  Hadoop: The Definitive Guide , 2009 .

[9]  Yogesh L. Simmhan,et al.  Cloud-Based Software Platform for Big Data Analytics in Smart Grids , 2013, Computing in Science & Engineering.

[10]  Carol L. Stimmel,et al.  Big Data Analytics Strategies for the Smart Grid , 2014 .

[11]  S. R,et al.  Data Mining with Big Data , 2017, 2017 11th International Conference on Intelligent Systems and Control (ISCO).

[12]  Gianmarco De Francisci Morales,et al.  Big Data Stream Learning with SAMOA , 2014, 2014 IEEE International Conference on Data Mining Workshop.

[13]  Amr A. Munshi,et al.  Cloud-based visual analytics for smart grids big data , 2016, 2016 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT).

[14]  Jingmin Li.  Design of real-time data analysis system based on Impala , 2014, 2014 IEEE Workshop on Advanced Research and Technology in Industry Applications (WARTIA).

[15]  J. Manyika.  Big data: The next frontier for innovation, competition, and productivity , 2011 .

[16]  Erik Meijer.  The world according to LINQ , 2011, CACM.

[17]  Hairong Kuang,et al.  The Hadoop Distributed File System , 2010, 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST).

[18]  Carlo Curino,et al.  Apache Hadoop YARN: yet another resource negotiator , 2013, SoCC.

[19]  Phil Blythe,et al.  Analysis of electric vehicle driver recharging demand profiles and subsequent impacts on the carbon content of electric vehicle trips , 2013 .

[20]  Gonzalo Mateos,et al.  Modeling and Optimization for Big Data Analytics: (Statistical) learning tools for our era of data deluge , 2014, IEEE Signal Processing Magazine.

[21]  P. Siano,et al.  Advances in information technology for Smart Grids , 2013, IECON 2013 - 39th Annual Conference of the IEEE Industrial Electronics Society.

[22]  Ehab Al-Shaer,et al.  Secure and Private Data Aggregation for Energy Consumption Scheduling in Smart Grids , 2017, IEEE Transactions on Dependable and Secure Computing.

[23]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[24]  F. Fakhar,et al.  Management of Symmetric Cryptographic Keys in cloud based environment , 2013, 2013 15th International Conference on Advanced Communications Technology (ICACT).

[25]  Nada Golmie,et al.  NIST Framework and Roadmap for Smart Grid Interoperability Standards, Release 1.0 , 2010 .

[26]  Gilbert M. Masters,et al.  Renewable and Efficient Electric Power Systems , 2004 .

[27]  John Gantz,et al.  The Digital Universe in 2020: Big Data, Bigger Digital Shadows, and Biggest Growth in the Far East , 2012 .

[28]  Qian Ai,et al.  A Random Matrix Theoretical Approach to Early Event Detection in Smart Grid , 2015 .

[29]  Yonggang Wen,et al.  Toward Scalable Systems for Big Data Analytics: A Technology Tutorial , 2014, IEEE Access.

[30]  Qian Ai,et al.  A Big Data Architecture Design for Smart Grids Based on Random Matrix Theory , 2015, IEEE Transactions on Smart Grid.

[31]  Jorge Bernardino,et al.  Big Data Issues , 2015, IDEAS.

[32]  Pete Wyckoff,et al.  Hive - A Warehousing Solution Over a Map-Reduce Framework , 2009, Proc. VLDB Endow..