Holistic Measures for Evaluating Prediction Models in Smart Grids

The performance of prediction models is often based on “abstract metrics” that estimate the model's ability to limit residual errors between the observed and predicted values. However, meaningful evaluation and selection of prediction models for end-user domains requires holistic and application-sensitive performance measures. Inspired by energy consumption prediction models used in the emerging “big data” domain of Smart Power Grids, we propose a suite of performance measures to rationally compare models along the dimensions of scale independence, reliability, volatility and cost. We include both application independent and dependent measures, the latter parameterized to allow customization by domain experts to fit their scenario. While our measures are generalizable to other domains, we offer an empirical analysis using real energy use data for three Smart Grid applications: planning, customer education and demand response, which are relevant for energy sustainability. Our results underscore the value of the proposed measures to offer a deeper insight into models' behavior and their impact on real applications, which benefit both data mining researchers and practitioners.

[1]  Remy Cottet,et al.  Bayesian Modeling and Forecasting of Intraday Electricity Load , 2003 .

[2]  Francisco Martinez Alvarez,et al.  Energy Time Series Forecasting Based on Pattern Sequence Similarity , 2011, IEEE Transactions on Knowledge and Data Engineering.

[3]  Hesham K. Alfares,et al.  Electric load forecasting: Literature survey and classification of methods , 2002, Int. J. Syst. Sci..

[4]  Nima Amjady,et al.  Short-term hourly load forecasting using time-series modeling with peak load estimation capability , 2001 .

[5]  Xinghuo Yu,et al.  The New Frontier of Smart Grids , 2011, IEEE Industrial Electronics Magazine.

[6]  Johanna L. Mathieu,et al.  Variability in automated responses of commercial buildings and industrial facilities to dynamic elec , 2011 .

[7]  Yogesh L. Simmhan,et al.  Improving Energy Use Forecast for Campus Micro-grids Using Indirect Indicators , 2011, 2011 IEEE 11th International Conference on Data Mining Workshops.

[8]  Manish Marwah,et al.  Fine-Grained Photovoltaic Output Prediction Using a Bayesian Ensemble , 2012, AAAI.

[9]  Luís Torgo,et al.  Precision and Recall for Regression , 2009, Discovery Science.

[10]  Sila Kiliccote,et al.  Estimating Demand Response Load Impacts: Evaluation of BaselineLoad Models for Non-Residential Buildings in California , 2008 .

[11]  Yogesh L. Simmhan,et al.  Toward data-driven demand-response optimization in a campus microgrid , 2011, BuildSys '11.

[12]  Gabriel H. Tucci,et al.  Analysis of the residential, commercial and industrial electricity consumption , 2011, 2011 IEEE PES Innovative Smart Grid Technologies.

[13]  Juan Shishido,et al.  Smart Meter Data Quality Insights , 2012 .

[14]  Kiri Wagstaff,et al.  Machine Learning that Matters , 2012, ICML.

[15]  Alireza Khotanzad,et al.  ANNSTLF-Artificial Neural Network Short-Term Load Forecaster- generation three , 1998 .

[16]  Gustavo E. A. P. A. Batista,et al.  A Survey on Graphical Methods for Classification Predictive Performance Evaluation , 2011, IEEE Transactions on Knowledge and Data Engineering.

[17]  Rob J Hyndman,et al.  Automatic Time Series Forecasting: The forecast Package for R , 2008 .

[18]  Ming-Wei Chang,et al.  Load Forecasting Using Support Vector Machines: A Study on EUNITE Competition 2001 , 2004, IEEE Transactions on Power Systems.

[19]  Sarvapali D. Ramchurn,et al.  Putting the 'smarts' into the smart grid , 2012, Commun. ACM.

[20]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[21]  R. Fildes,et al.  Measuring forecasting accuracy : the case of judgmental adjustments to SKU-level demand forecasts , 2013 .

[22]  Oliver Kramer,et al.  Power Prediction in Smart Grids with Evolutionary Local Kernel Regression , 2010, HAIS.

[23]  Ly Fie Sugianto,et al.  Demand forecasting in the deregulated market: a bibliography survey , 2002 .

[24]  Clive W. J. Granger,et al.  Prediction with a generalized cost of error function , 1969 .

[25]  P. Young,et al.  Time series analysis, forecasting and control , 1972, IEEE Transactions on Automatic Control.

[26]  G. Kenny,et al.  Forecasting irish inflation using ARIMA models , 1998 .

[27]  Fred L. Collopy,et al.  Error Measures for Generalizing About Forecasting Methods: Empirical Comparisons , 1992 .

[28]  Stan Szpakowicz,et al.  Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation , 2006, Australian Conference on Artificial Intelligence.

[29]  Bart Baesens,et al.  A Novel Profit Maximizing Metric for Measuring Classification Performance of Customer Churn Prediction Models , 2013, IEEE Transactions on Knowledge and Data Engineering.

[30]  B. Dong,et al.  Applying support vector machines to predict building energy consumption in tropical region , 2005 .

[31]  Rob J Hyndman,et al.  Another look at measures of forecast accuracy , 2006 .

[32]  Aimee McKane.  Automated Demand Response: The Missing Link in the Electricity Value Chain , 2010 .

[33]  Yogesh L. Simmhan,et al.  Cloud-Based Software Platform for Big Data Analytics in Smart Grids , 2013, Computing in Science & Engineering.

[34]  Peter Grindrod,et al.  A new error measure for forecasts of household-level, high resolution electrical energy consumption , 2014 .

[35]  Preprint Mps,et al.  A New Error Measure for Forecasts of Household-level, High Resolution Electrical Energy Consumption , 2012 .

[36]  Hiroyuki Mori,et al.  Hybrid intelligent method of relevant vector machine and regression tree for probabilistic load forecasting , 2011, 2011 2nd IEEE PES International Conference and Exhibition on Innovative Smart Grid Technologies.

[37]  M. McIntyre,et al.  Analysis of forecasting algorithms for minimization of electric demand costs for electric vehicle charging in commercial and industrial environments , 2012, 2012 IEEE Transportation Electrification Conference and Expo (ITEC).