A Hybrid ICT-Solution for Smart Meter Data Analytics

Smart meters are increasingly used worldwide. Smart meters are the advanced meters capable of measuring energy consumption at a fine-grained time interval, e.g., every 15 min. Smart meter data are typically bundled with social economic data in analytics, such as meter geographic locations, weather conditions and user information, which makes the data sets very sizable and the analytics complex. Data mining and emerging cloud computing technologies make collecting, processing, and analyzing the so-called big data possible. This paper proposes an innovative ICT-solution to streamline smart meter data analytics. The proposed solution offers an information integration pipeline for ingesting data from smart meters, a scalable platform for processing and mining big data sets, and a web portal for visualizing analytics results. The implemented system has a hybrid architecture of using Spark or Hive for big data processing, and using the machine learning toolkit, MADlib, for doing in-database data analytics in PostgreSQL database. This paper evaluates the key technologies of the proposed ICT-solution, and the results show the effectiveness and efficiency of using the system for both batch and online analytics.

[1]  Torben Bach Pedersen,et al.  MapReduce-based Dimensional ETL Made Easy , 2012, Proc. VLDB Endow..

[2]  Z. Vale,et al.  An electric energy consumer characterization framework based on data mining techniques , 2005, IEEE Transactions on Power Systems.

[3]  Torben Bach Pedersen,et al.  CloudETL: scalable dimensional ETL for hive , 2014, IDEAS.

[4]  Silvia Santini,et al.  Revealing Household Characteristics from Smart Meter Data , 2014 .

[5]  Torben Bach Pedersen,et al.  ETLMR: A Highly Scalable Dimensional ETL Framework Based on MapReduce , 2013, Trans. Large Scale Data Knowl. Centered Syst..

[6]  Manish Marwah,et al.  IoTAbench: an Internet of Things Analytics Benchmark , 2015, ICPE.

[7]  Ram Rajagopal,et al.  Building dynamic thermal profiles of energy consumption for individuals and neighborhoods , 2013, 2013 IEEE International Conference on Big Data.

[8]  Lukasz Golab,et al.  SMAS: A smart meter data analytics system , 2015, 2015 IEEE 31st International Conference on Data Engineering.

[9]  Tilmann Rabl,et al.  DGFIndex for Smart Grid: Enhancing Hive with a Cost-Effective Multidimensional Range Index , 2014, Proc. VLDB Endow..

[10]  Karl Aberer,et al.  SmartD: smart meter data analytics dashboard , 2014, e-Energy.

[11]  G. Chicco,et al.  Comparisons among clustering techniques for electricity customer classification , 2006, IEEE Transactions on Power Systems.

[12]  Wolfgang Lehner,et al.  SAP HANA database: data management for modern business applications , 2012, SGMD.

[13]  Rodney Anthony Stewart,et al.  Web-based knowledge management system: linking smart metering to the future of urban water planning , 2010 .

[14]  Rodney Anthony Stewart,et al.  Development of a three-phase battery energy storage scheduling and operation system for low voltage distribution networks , 2015 .

[15]  B. De Moor,et al.  Short-term load forecasting, profile identification, and customer segmentation: a methodology based on periodic time series , 2005, IEEE Transactions on Power Systems.

[16]  Francisco C. Pereira,et al.  Using pattern recognition to identify habitual behavior in residential electricity consumption , 2012 .

[17]  Mikko Kolehmainen,et al.  Data-based method for creating electricity use load profiles using large amount of customer-specific hourly measured electricity use data , 2010 .

[18]  M. Etezadi-Amoli,et al.  Smart meter based short-term load forecasting for residential customers , 2011, 2011 North American Power Symposium.

[19]  Eamonn J. Keogh,et al.  On the Need for Time Series Data Mining Benchmarks: A Survey and Empirical Demonstration , 2002, Data Mining and Knowledge Discovery.

[20]  Rodney Anthony Stewart,et al.  Forecasting low voltage distribution network demand profiles using a pattern recognition based expert system , 2014 .

[21]  Xiufeng Liu,et al.  Streamlining Smart Meter Data Analytics , 2015 .

[22]  Scott Shenker,et al.  Discretized Streams: An Efficient and Fault-Tolerant Model for Stream Processing on Large Clusters , 2012, HotCloud.

[23]  N.D. Hatziargyriou,et al.  Two-Stage Pattern Recognition of Load Curves for Classification of Electricity Customers , 2007, IEEE Transactions on Power Systems.

[24]  Lili Zuo,et al.  Predicting energy consumption of multiproduct pipeline using artificial neural networks , 2014 .

[25]  Junwei Lu,et al.  Autoregressive with Exogenous Variables and Neural Network Short-Term Load Forecast Models for Residential Low Voltage Distribution Networks , 2014 .

[26]  Ian Beausoleil-Morrison,et al.  Disaggregating categories of electrical energy end-use from whole-house hourly data , 2012 .

[27]  Zheng Shao,et al.  Hive - a petabyte scale data warehouse using Hadoop , 2010, 2010 IEEE 26th International Conference on Data Engineering (ICDE 2010).

[28]  Rayman Preet Singh,et al.  Computing Electricity Consumption Profiles from Household Smart Meter Data , 2014, EDBT/ICDT Workshops.

[29]  Can Anil,et al.  Benchmarking of Data Mining Techniques as Applied to Power System Analysis , 2013 .

[30]  Rodney Anthony Stewart,et al.  Intelligent autonomous system for residential water end use classification: Autoflow , 2015, Appl. Soft Comput..

[31]  Wojciech M. Golab,et al.  Benchmarking Smart Meter Data Analytics , 2015, EDBT.

[32]  Ram Rajagopal,et al.  Smart Meter Driven Segmentation: What Your Consumption Says About You , 2013, IEEE Transactions on Power Systems.

[33]  Stéphane Ploix,et al.  Prediction of appliances energy use in smart homes , 2012 .

[34]  Enrique Personal,et al.  Key performance indicators: A useful tool to assess Smart Grid goals , 2014 .