Probabilistic electric load forecasting through Bayesian Mixture Density Networks

Probabilistic load forecasting (PLF) is a key component in the extended toolchain required for efficient management of smart energy grids. Neural networks are widely considered to achieve improved prediction performances, supporting highly flexible mappings of complex relationships between the target and the conditioning variables set. However, obtaining comprehensive predictive uncertainties from such black-box models is still a challenging and unsolved problem. In this work, we propose a novel PLF approach, framed on Bayesian Mixture Density Networks. Both aleatoric and epistemic uncertainty sources are encompassed within the model predictions, inferring general conditional densities, depending on the input features, within an end-to-end training framework. To achieve reliable and computationally scalable estimators of the posterior distributions, both Mean Field variational inference and deep ensembles are integrated. Experiments have been performed on household short-term load forecasting tasks, showing the capability of the proposed method to achieve robust performances in different operating conditions.

[1]  Yee Whye Teh,et al.  Bayesian Learning via Stochastic Gradient Langevin Dynamics , 2011, ICML.

[2]  Dahua GAN,et al.  Embedding based quantile regression neural network for probabilistic load forecasting , 2018 .

[3]  Sebastian Nowozin,et al.  How Good is the Bayes Posterior in Deep Neural Networks Really? , 2020, ICML.

[4]  P. Gaillard,et al.  Additive models and robust aggregation for GEFCom2014 probabilistic electric load and electricity price forecasting , 2016 .

[5]  Kyungjae Lee,et al.  Uncertainty-Aware Learning from Demonstration Using Mixture Density Networks with Sampling-Free Variance Modeling , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[6]  A Survey on Bayesian Deep Learning , 2020, ACM Comput. Surv..

[7]  R. Weron Modeling and Forecasting Electricity Loads and Prices: A Statistical Approach , 2006 .

[8]  Ming-Wei Chang,et al.  Load Forecasting Using Support Vector Machines: A Study on EUNITE Competition 2001 , 2004, IEEE Transactions on Power Systems.

[9]  Yi Wang,et al.  Using Bayesian Deep Learning to Capture Uncertainty for Residential Net Load Forecasting , 2020, IEEE Transactions on Power Systems.

[10]  Andrew Gordon Wilson,et al.  Bayesian Deep Learning and a Probabilistic Perspective of Generalization , 2020, NeurIPS.

[11]  Pramod Viswanath,et al.  Breaking the Bandwidth Barrier: Geometrical Adaptive Entropy Estimation , 2016, IEEE Transactions on Information Theory.

[12]  Sahin Albayrak,et al.  Short-Term Probabilistic Load Forecasting at Low Aggregation Levels Using Convolutional Neural Networks , 2019, 2019 IEEE Milan PowerTech.

[13]  Yi Wang,et al.  Review of Smart Meter Data Analytics: Applications, Methodologies, and Challenges , 2018, IEEE Transactions on Smart Grid.

[14]  Bart De Schutter,et al.  Forecasting spot electricity prices Deep learning approaches and empirical comparison of traditional algorithms , 2018 .

[15]  A. Raftery,et al.  Probabilistic forecasts, calibration and sharpness , 2007 .

[16]  Alexander Jordan,et al.  Evaluating Probabilistic Forecasts with scoringRules , 2017, Journal of Statistical Software.

[17]  Yee Whye Teh,et al.  Neural Ensemble Search for Performant and Calibrated Predictions , 2020, ArXiv.

[18]  Yandong Yang,et al.  Power load probability density forecasting using Gaussian process quantile regression , 2017 .

[19]  Sebastian Nowozin,et al.  Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift , 2019, NeurIPS.

[20]  Silja Meyer-Nieberg,et al.  Electric load forecasting methods: Tools for decision making , 2009, Eur. J. Oper. Res..

[21]  J. Landes,et al.  Strictly Proper Scoring Rules , 2014 .

[22]  Ignacio E. Grossmann,et al.  Optimal scheduling of industrial combined heat and power plants under time-sensitive electricity prices , 2013 .

[23]  Maksim Terpilowski,et al.  scikit-posthocs: Pairwise multiple comparison tests in Python , 2019, J. Open Source Softw..

[24]  Jim Tørresen,et al.  RoboJam: A Musical Mixture Density Network for Collaborative Touchscreen Interaction , 2017, EvoMUSART.

[25]  Thomas P. Minka,et al.  Bayesian model averaging is not model combination , 2002 .

[26]  Iiro Harjunkoski,et al.  Stochastic short-term integrated electricity procurement and production scheduling for a large consumer , 2021, Comput. Chem. Eng..

[27]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[28]  Patricia J. Culligan,et al.  Short-term apartment-level load forecasting using a modified neural network with selected auto-regressive features , 2021, Applied Energy.

[29]  Alex Graves,et al.  Generating Sequences With Recurrent Neural Networks , 2013, ArXiv.

[30]  Shanlin Yang,et al.  A deep learning model for short-term power load and probability density forecasting , 2018, Energy.

[31]  S. Srihari Mixture Density Networks , 1994 .

[32]  Yarin Gal,et al.  Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations , 2020, NeurIPS.

[33]  M. Pantos,et al.  Short-Term Transmission-Loss Forecast for the Slovenian Transmission Power System Based on a Fuzzy-Logic Decision Approach , 2011, IEEE Transactions on Power Systems.

[34]  Joakim Widén,et al.  Very short term load forecasting of residential electricity consumption using the Markov-chain mixture distribution (MCM) model , 2021 .

[35]  Chiwoo Park,et al.  Short-term electric load forecasting for buildings using logistic mixture vector autoregressive model with curve registration , 2021 .

[36]  Andrea Vitali,et al.  Bayesian deep learning based method for probabilistic forecast of day-ahead electricity prices , 2019, Applied Energy.

[37]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[38]  Rainer Göb,et al.  Electrical load forecasting by exponential smoothing with covariates , 2013 .

[39]  Tianqi Chen,et al.  Stochastic Gradient Hamiltonian Monte Carlo , 2014, ICML.

[40]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[41]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[42]  Song Li,et al.  A Novel Wavelet-Based Ensemble Method for Short-Term Load Forecasting with Hybrid Neural Networks and Feature Selection , 2016, IEEE Transactions on Power Systems.

[43]  Jun Hu,et al.  Short-Term Load Forecasting With Deep Residual Networks , 2018, IEEE Transactions on Smart Grid.

[44]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[45]  장윤희,et al.  Y. , 2003, Industrial and Labor Relations Terms.

[46]  Dipti Srinivasan,et al.  An Improved Quantile Regression Neural Network for Probabilistic Load Forecasting , 2019, IEEE Transactions on Smart Grid.

[47]  R. Weron,et al.  Recent advances in electricity price forecasting: A review of probabilistic forecasting , 2016 .

[48]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[49]  Ching-Hsin Wang,et al.  Enhanced Gaussian process mixture model for short-term electric load forecasting , 2019, Inf. Sci..

[50]  Stefano Ermon,et al.  Accurate Uncertainties for Deep Learning Using Calibrated Regression , 2018, ICML.

[51]  Ning Zhang,et al.  Probabilistic individual load forecasting using pinball loss guided LSTM , 2019, Applied Energy.

[52]  Christian P. Robert,et al.  Handbook of Mixture Analysis , 2018 .

[53]  Thomas B. Schön,et al.  Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[54]  J. Ben Hadj Slama,et al.  Day-ahead load forecast using random forest and expert input selection , 2015 .

[55]  Joelle Pineau,et al.  Handling Black Swan Events in Deep Learning with Diversely Extrapolated Neural Networks , 2020, IJCAI.

[56]  Finale Doshi-Velez,et al.  Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning , 2017, ICML.

[57]  A. Kraskov,et al.  Estimating mutual information. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[58]  Carlos Dias Maciel,et al.  Multiple households very short-term load forecasting using bayesian networks , 2020 .

[59]  Miss A.O. Penney (b) , 1974, The New Yale Book of Quotations.

[60]  Tao Hong,et al.  Probabilistic electric load forecasting: A tutorial review , 2016 .

[61]  Ian T. Nabney,et al.  Regularisation of mixture density networks , 1999 .

[62]  Mohammad Rastegar,et al.  Deep-Based Conditional Probability Density Function Forecasting of Residential Loads , 2020, IEEE Transactions on Smart Grid.

[63]  D. Ramin,et al.  Demand-side management via optimal production scheduling in power-intensive industries: The case of metal casting process , 2018, Applied Energy.

[64]  Yi Wang,et al.  Combining Probability Density Forecasts for Power Electrical Loads , 2020, IEEE Transactions on Smart Grid.

[65]  Weijun Hong,et al.  Deep ensemble learning based probabilistic load forecasting in smart grids , 2019 .

[66]  Geoffrey E. Hinton,et al.  Keeping the neural networks simple by minimizing the description length of the weights , 1993, COLT '93.

[67]  A. Raftery,et al.  Using Bayesian Model Averaging to Calibrate Forecast Ensembles , 2005 .

[68]  Lei Xu,et al.  Probabilistic load forecasting for buildings considering weather forecasting uncertainty and uncertain peak load , 2019, Applied Energy.

[69]  Yi Wang,et al.  Load probability density forecasting by transforming and combining quantile forecasts , 2020, Applied Energy.

[70]  Danna Zhou,et al.  d. , 1840, Microbial pathogenesis.

[71]  Chao Wang,et al.  Electricity consumption probability density forecasting method based on LASSO-Quantile Regression Neural Network , 2019, Applied Energy.

[72]  Andrew Gordon Wilson,et al.  Subspace Inference for Bayesian Deep Learning , 2019, UAI.

[73]  M. Wand,et al.  EXACT MEAN INTEGRATED SQUARED ERROR , 1992 .

[74]  Dustin Tran,et al.  Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches , 2018, ICLR.

[75]  Matthias Weigold,et al.  Machine learning based very short term load forecasting of machine tools , 2020 .

[76]  Julien Cornebise,et al.  Weight Uncertainty in Neural Networks , 2015, ArXiv.

[77]  Thomas Brox,et al.  Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[78]  Souhaib Ben Taieb,et al.  A gradient boosting approach to the Kaggle load forecasting competition , 2014 .

[79]  Leandro dos Santos Coelho,et al.  Enhanced ensemble structures using wavelet neural networks applied to short-term load forecasting , 2019, Eng. Appl. Artif. Intell..

[80]  Kilian Q. Weinberger,et al.  On Calibration of Modern Neural Networks , 2017, ICML.

[81]  Li Mo,et al.  Day-ahead short-term load probability density forecasting method with a decomposition-based quantile regression forest , 2020 .

[82]  Michael A. Osborne,et al.  Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning , 2019, AISTATS.

[83]  Soumya Ghosh,et al.  Quality of Uncertainty Quantification for Bayesian Neural Network Inference , 2019, ArXiv.