Compression of smart meter big data: A survey

Abstract In recent years, the smart grid has attracted wide attention from around the world. Large scale data are collected by sensors and measurement devices in a smart grid. Smart meters can record fine-grained information about electricity consumption in near real-time, thus forming the smart meter big data. Smart meter big data has provided new opportunities for electric load forecasting, anomaly detection, and demand side management. However, the high-dimensional and massive smart meter big data not only creates great pressure on data transmission lines, but also incur enormous storage costs on data centres. Therefore, to reduce the transmission pressure and storage overhead, improve data mining efficiency, and thus fulfil the potential of smart meter big data. This study presents a comprehensive study on the compression techniques for smart meter big data. The development of smart grids and the characteristics and application challenges of electric power big data are first introduced, followed by analysis of the characteristics and benefits of smart meter big data. Finally, this study focuses on the potential data compression methods for smart meter big data, and discusses the evaluation methods for smart meter big data compression.

[1]  Wai Ming To,et al.  The causal relationship between electricity consumption and economic growth in a Gaming and Tourism Center: The case of Macao SAR, the People’s Republic of China , 2010, Energy.

[2]  Cong Liu,et al.  A Wavelet-Based Data Compression Technique for Smart Grid , 2011, IEEE Transactions on Smart Grid.

[3]  Brandon J. Murrill,et al.  Smart Meter Data: Privacy and Cybersecurity , 2012 .

[4]  Young-Jin Kim,et al.  A secure decentralized data-centric information infrastructure for smart grid , 2010, IEEE Communications Magazine.

[5]  Anna Scaglione,et al.  For the Grid and Through the Grid: The Role of Power Line Communications in the Smart Grid , 2010, Proceedings of the IEEE.

[6]  Tim Littler,et al.  Wavelets for the analysis and compression of power system disturbances , 1999 .

[7]  J. A. Aguado,et al.  Smart metering trends, implications and necessities: A policy review , 2016 .

[8]  Chongqing Kang,et al.  Smart Metering Load Data Compression Based on Load Feature Identification , 2016, IEEE Transactions on Smart Grid.

[9]  Daniel Enrique Lucani,et al.  Network Coding Protocols for Smart Grid Communications , 2014, IEEE Transactions on Smart Grid.

[10]  J. Kraus,et al.  Optimal data compression techniques for Smart Grid and power quality trend data , 2012, 2012 IEEE 15th International Conference on Harmonics and Quality of Power.

[11]  Tanveer Ahmad,et al.  Non-technical loss analysis and prevention using smart meters , 2017 .

[12]  Shanlin Yang,et al.  Understanding household energy consumption behavior: The contribution of energy big data analytics , 2016 .

[13]  Divesh Srivastava,et al.  Data quality: The other face of Big Data , 2014, 2014 IEEE 30th International Conference on Data Engineering.

[14]  Ken J. Nixon,et al.  Low-loss, high-compression of energy profiles , 2015, 2015 International Conference on Renewable Energy Research and Applications (ICRERA).

[15]  C. L. Anderson,et al.  Development of a Hybrid Model for Electrical Power Spot Prices , 2002, IEEE Power Engineering Review.

[16]  George K. Karagiannidis,et al.  Big Data Analytics for Dynamic Energy Management in Smart Grids , 2015, Big Data Res..

[17]  Donald E. Knuth,et al.  Dynamic Huffman Coding , 1985, J. Algorithms.

[18]  Michael Conlon,et al.  A clustering approach to domestic electricity load profile characterisation using smart metering data , 2015 .

[19]  M. Ringwelski,et al.  The Hitchhiker's guide to choosing the compression algorithm for your smart meter data , 2012, 2012 IEEE International Energy Conference and Exhibition (ENERGYCON).

[20]  Mo Yuanbin,et al.  A Data Compression Algorithm Based on Adaptive Huffman Code for Wireless Sensor Networks , 2011, 2011 Fourth International Conference on Intelligent Computation Technology and Automation.

[21]  Fei Gao,et al.  Density prediction and dimensionality reduction of mid-term electricity demand in China: A new semiparametric-based additive model , 2014 .

[22]  Klemens Böhm,et al.  A time-series compression technique and its application to the smart grid , 2014, The VLDB Journal.

[23]  Shen Jie.  Survey on Smart Grid Technology , 2009 .

[24]  Matti Lehtonen,et al.  Clustering of Connection Points and Load Modeling in Distribution Systems , 2013, IEEE Transactions on Power Systems.

[25]  Shanlin Yang,et al.  Big data driven smart energy management: From big data to big insights , 2016 .

[26]  Xi Fang,et al.  3. Full Four-channel 6.3-gb/s 60-ghz Cmos Transceiver with Low-power Analog and Digital Baseband Circuitry 7. Smart Grid — the New and Improved Power Grid: a Survey , 2022 .

[27]  Ilhami Colak,et al.  Smart grid technologies and applications , 2016 .

[28]  Jun Hasegawa,et al.  Dynamic state estimation including anomaly detection and identification for power systems , 1982 .

[29]  A. Notaristefano,et al.  Data size reduction with symbolic aggregate approximation for electrical load pattern grouping , 2013 .

[30]  Shanlin Yang,et al.  Optimal load distribution model of microgrid in the smart grid environment , 2014 .

[31]  Gonzalo Navarro,et al.  Self-indexing Based on LZ77 , 2011, CPM.

[32]  S. Sharma.  The relationship between energy and economic growth: Empirical evidence from 66 countries , 2010 .

[33]  S.N. Singh,et al.  An Approach for Optimal Placement of Static VAr Compensators Based on Reactive Power Spot Price , 2007, IEEE Transactions on Power Systems.

[34]  Víctor Manuel Fernandes Mendes,et al.  Classification of new electricity customers based on surveys and smart metering data , 2016 .

[35]  Stuart E. Madnick,et al.  Improving data quality through effective use of data semantics , 2006, Data Knowl. Eng..

[36]  Abhisek Ukil,et al.  Automated analysis of power systems disturbance records: Smart Grid big data perspective , 2014, 2014 IEEE Innovative Smart Grid Technologies - Asia (ISGT ASIA).

[37]  Friedemann Mattern,et al.  Leveraging smart meter data to recognize home appliances , 2012, 2012 IEEE International Conference on Pervasive Computing and Communications.

[38]  Ling Wang,et al.  Development of Low-Carbon Electricity System Based on Smart Grid , 2015, 2015 8th International Conference on Control and Automation (CA).

[39]  Farshid Keynia,et al.  Short-term load forecasting of power systems by combination of wavelet transform and neuro-evolutionary algorithm , 2009 .

[40]  Stéphane Mallat,et al.  A Theory for Multiresolution Signal Decomposition: The Wavelet Representation , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  Christos A. Papachristou,et al.  Multiscan-based test compression and hardware decompression using LZ77 , 2002, Proceedings. International Test Conference.

[42]  Conversion and delivery of electrical energy in the 21st century , 2008, IEEE Power & Energy Society General Meeting.

[43]  W. L. Kling,et al.  Day-ahead residential load forecasting with artificial neural networks using smart meter data , 2013, 2013 IEEE Grenoble Conference.

[44]  Michel Barlaud,et al.  Image coding using wavelet transform , 1992, IEEE Trans. Image Process..

[45]  Jila-Ayubi.  Lossy Color Image Compression Based on Singular Value Decomposition and GNU GZIP , 2014 .

[46]  S. Santoso,et al.  Power quality disturbance data compression using wavelet transform methods , 1997 .

[47]  Mária Bieliková,et al.  Symbolic Time Series Representation for Stream Data Processing , 2015, 2015 IEEE Trustcom/BigDataSE/ISPA.

[48]  C. Tang.  A re-examination of the relationship between electricity consumption and economic growth in Malaysia , 2008 .

[49]  B.F. Wollenberg,et al.  Toward a smart grid: power delivery for the 21st century , 2005, IEEE Power and Energy Magazine.

[50]  Francesco Benzi,et al.  Electricity Smart Meters Interfacing the Households , 2011, IEEE Transactions on Industrial Electronics.

[51]  Yasin Kabalci,et al.  A survey on smart metering and smart grid communication , 2016 .

[52]  S. Brindha,et al.  Power quality monitoring and compression using the discrete wavelet transform , 2013, 2013 International Conference on Advanced Computing and Communication Systems.

[53]  Sung-Kwan Joo,et al.  Holiday Load Forecasting Using Fuzzy Polynomial Regression With Weather Feature Selection and Adjustment , 2012, IEEE Transactions on Power Systems.

[54]  Tapas K. Mallick,et al.  A review of Integration, Control, Communication and Metering (ICCM) of renewable energy based smart grid , 2014 .

[55]  J. Schleich,et al.  Persistence of the effects of providing feedback alongside smart metering devices on household electricity demand , 2017 .

[56]  Chao Shen,et al.  A review of electric load classification in smart grid environment , 2013 .

[57]  N. Voropai.  Investments and development of electric power industry in market environment , 2002, Proceedings. International Conference on Power System Technology.

[58]  H. Farhangi,et al.  The path of the smart grid , 2010, IEEE Power and Energy Magazine.

[59]  Sharad Mehrotra,et al.  Capturing sensor-generated time series with quality guarantees , 2003, Proceedings 19th International Conference on Data Engineering (Cat. No.03CH37405).

[60]  Shijie Cheng,et al.  Study of the key technologies of electric power big data and its application prospects in smart grid , 2014, 2014 IEEE PES Asia-Pacific Power and Energy Engineering Conference (APPEEC).

[61]  Karl Aberer,et al.  Symbolic representation of smart meter data , 2013, EDBT '13.

[62]  梁波,et al.  Computation Services and Applications of Electricity Big Data Based on Hadoop , 2015 .

[63]  Klemens Böhm,et al.  Re-identification of Smart Meter data , 2012, Personal and Ubiquitous Computing.

[64]  D. Sibley.  Spot Pricing of Electricity , 1990 .

[65]  Nizar Zorba,et al.  Performance evaluation of power demand scheduling scenarios in a smart grid environment , 2015 .

[66]  Leandros Tassiulas,et al.  Smart grid technologies for future radio and data center networks , 2014, IEEE Communications Magazine.

[67]  Tin Kam Ho,et al.  A Sparse Coding Approach to Household Electricity Demand Forecasting in Smart Grids , 2017, IEEE Transactions on Smart Grid.

[68]  Z. Zakaria,et al.  Two-stage fuzzy clustering approach for load profiling , 2009, 2009 44th International Universities Power Engineering Conference (UPEC).

[69]  Thierry Blu,et al.  Linear interpolation revitalized , 2004, IEEE Transactions on Image Processing.

[70]  John S. Thompson,et al.  Impact of compression and aggregation in wireless networks on smart meter data , 2016, 2016 IEEE 17th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC).

[71]  Dusit Niyato,et al.  Cooperative transmission for meter data collection in smart grid , 2012, IEEE Communications Magazine.

[72]  G. Lambert-Torres,et al.  Anomaly detection in power system control center critical infrastructures using rough classification algorithm , 2009, 2009 3rd IEEE International Conference on Digital Ecosystems and Technologies.

[73]  Wang Xiangwe.  Computation Services and Applications of Electricity Big Data Based on Hadoop , 2015 .

[74]  Thomas J Overbye,et al.  Key Technical Challenges for the Electric Power Industry and Climate Change , 2010, IEEE Transactions on Energy Conversion.

[75]  D. Aytaç,et al.  The Relationship between Electricity Consumption, Electricity Price and Economic Growth in Turkey: 1984-2007 , 2011 .

[76]  Duncan S. Callaway,et al.  Using smart meter data to estimate demand response potential, with application to solar energy integration , 2014 .

[77]  Pierluigi Siano,et al.  Demand response and smart grids—A survey , 2014 .

[78]  Liusheng Huang,et al.  A privacy preserving scheme for change increment of electricity big data base on symmetric cryptography , 2014, 2014 IEEE 3rd International Conference on Cloud Computing and Intelligence Systems.

[79]  G. Chicco,et al.  Comparisons among clustering techniques for electricity customer classification , 2006, IEEE Transactions on Power Systems.

[80]  João Marcos Travassos Romano,et al.  The Compression of Electric Signal Waveforms for Smart Grids: State of the Art and Future Trends , 2014, IEEE Transactions on Smart Grid.

[81]  R. Mehra,et al.  Analysis of PCA based compression and denoising of smart grid data under normal and fault conditions , 2013, 2013 IEEE International Conference on Electronics, Computing and Communication Technologies.

[82]  Tan Zhong-fu,et al.  How to develop distributed generation in China: In the context of the reformation of electric power system , 2016 .

[83]  Xiaoxin Zhou,et al.  Knowledge model for electric power big data based on ontology and semantic web , 2015 .

[84]  Shanlin Yang,et al.  Demand side management in China: The context of China’s power industry reform , 2015 .

[85]  Prashant J. Shenoy,et al.  Private memoirs of a smart meter , 2010, BuildSys '10.

[86]  Z. Jane Wang,et al.  Home Appliance Load Modeling From Aggregated Smart Meter Data , 2015, IEEE Transactions on Power Systems.

[87]  Jianzhong Wu,et al.  State estimation of medium voltage distribution networks using smart meter measurements , 2016 .

[88]  Taskin Koçak,et al.  Smart Grid Technologies: Communication Technologies and Standards , 2011, IEEE Transactions on Industrial Informatics.

[89]  Lingfeng Wang,et al.  Smart meters for power grid — Challenges, issues, advantages and status , 2011, 2011 IEEE/PES Power Systems Conference and Exposition.

[90]  A. Laub,et al.  The singular value decomposition: Its computation and some applications , 1980 .

[91]  Paul Mangiameli,et al.  The Effects and Interactions of Data Quality and Problem Complexity on Classification , 2011, JDIQ.

[92]  Silvia Santini,et al.  Revealing Household Characteristics from Smart Meter Data , 2014 .

[93]  Gene H. Golub,et al.  Singular value decomposition and least squares solutions , 1970, Milestones in Matrix Computation.

[94]  Terry A. Welch,et al.  A Technique for High-Performance Data Compression , 1984, Computer.

[95]  Peter Mark Jansson,et al.  The role of the electric power industry in global sustainable development , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[96]  Shanlin Yang,et al.  Energy conservation and emission reduction of China’s electric power industry , 2015 .

[97]  Chongqing Kang,et al.  Sparse and Redundant Representation-Based Smart Meter Data Compression and Pattern Extraction , 2017, IEEE Transactions on Power Systems.

[98]  Eamonn J. Keogh,et al.  A symbolic representation of time series, with implications for streaming algorithms , 2003, DMKD '03.

[99]  Yi Wang,et al.  Clustering of Electricity Consumption Behavior Dynamics Toward Big Data Applications , 2016, IEEE Transactions on Smart Grid.

[100]  Nai-Kuan Chou,et al.  ECG data compression using truncated singular value decomposition , 2001, IEEE Trans. Inf. Technol. Biomed..

[101]  Yi Sun,et al.  Data compression and reconstruction of smart grid customers based on compressed sensing theory , 2016 .

[102]  Li Li,et al.  Leveraging smart meter data for economic optimization of residential photovoltaics under existing tariff structures and incentive schemes , 2017 .

[103]  Fei Jiang,et al.  Big data issues in smart grid – A review , 2017 .

[104]  Xiufeng Liu,et al.  A Hybrid ICT-Solution for Smart Meter Data Analytics , 2016, ArXiv.

[105]  Rowayda A. Sadek,et al.  SVD Based Image Processing Applications: State of The Art, Contributions and Research Challenges , 2012, ArXiv.

[106]  Awang Zainudin,et al.  Development of Portable Solar Electricity Generating System , 2014 .

[107]  Gregory Murphy,et al.  PMU data analysis in smart grid using WPD , 2014, IECON 2014 - 40th Annual Conference of the IEEE Industrial Electronics Society.

[108]  Zheng Haiya.  Analysis Technology and Typical Scenario Application of Electricity Big Data of Power Consumers , 2015 .

[109]  Gholamreza Anbarjafari,et al.  Lossy medical image compression using Huffman coding and singular value decomposition , 2013, 2013 21st Signal Processing and Communications Applications Conference (SIU).

[110]  Haiwang Zhong,et al.  Coupon incentive-based demand response (CIDR) in smart grid , 2012, 2012 IEEE Power and Energy Society General Meeting.

[111]  Christos Faloutsos,et al.  Fast Time Sequence Indexing for Arbitrary Lp Norms , 2000, VLDB.

[112]  P. McSharry,et al.  Short-Term Load Forecasting Methods: An Evaluation Based on European Data , 2007, IEEE Transactions on Power Systems.

[113]  Emmanouil Kalligeros,et al.  Optimal Selective Huffman Coding for Test-Data Compression , 2007, IEEE Transactions on Computers.

[114]  Jamshed Iqbal,et al.  Opportunities and challenges in control of smart grids – Pakistani perspective , 2017 .

[115]  Mengmeng Yu,et al.  Supply–demand balancing for power management in smart grid: A Stackelberg game approach , 2016 .

[116]  Julio Cesar Stacchini de Souza,et al.  Data Compression in Smart Distribution Systems via Singular Value Decomposition , 2017, IEEE Transactions on Smart Grid.

[117]  Yang Liu,et al.  Short-Term Load Forecasting Based on Big Data Technologies , 2014, CIT 2014.