Deep Distribution Regression

Due to their flexibility and predictive performance, machine-learning based regression methods have become an important tool for predictive modeling and forecasting. However, most methods focus on estimating the conditional mean or specific quantiles of the target quantity and do not provide the full conditional distribution, which contains uncertainty information that might be crucial for decision making. In this article, we provide a general solution by transforming a conditional distribution estimation problem into a constrained multi-class classification problem, in which tools such as deep neural networks. We propose a novel joint binary cross-entropy loss function to accomplish this goal. We demonstrate its performance in various simulation studies comparing to state-of-the-art competing methods. Additionally, our method shows improved accuracy in a probabilistic solar energy forecasting problem.

[1]  James W. Taylor.  A Quantile Regression Neural Network Approach to Estimating the Conditional Density of Multiperiod Returns , 2000 .

[2]  Joakim Widén,et al.  Review on probabilistic forecasting of photovoltaic power production and electricity consumption , 2018 .

[3]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[4]  Allan Timmermann,et al.  Density forecasting in economics and finance , 2000 .

[5]  Q. Shao,et al.  On Parameters of Increasing Dimensions , 2000 .

[6]  Ann B. Lee,et al.  Nonparametric Conditional Density Estimation in a High-Dimensional Regression Setting , 2016, 1604.00540.

[7]  Misha Pavel,et al.  Density Boosting for Gaussian Mixtures , 2004, ICONIP.

[8]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[9]  Amir F. Atiya,et al.  Comprehensive Review of Neural Network-Based Prediction Intervals and New Advances , 2011, IEEE Transactions on Neural Networks.

[10]  Furno Marilena,et al.  Quantile Regression , 2018, Wiley Series in Probability and Statistics.

[11]  Christopher R. Genovese,et al.  Conditional Density Estimation using Finite Mixture Models with an Application to Astrophysics , 2005 .

[12]  Durga L. Shrestha,et al.  Machine learning approaches for estimation of prediction interval for the model output , 2006, Neural Networks.

[13]  Rob J. Hyndman,et al.  Nonparametric Estimation and Symmetry Tests for Conditional Density Functions , 2002 .

[14]  Olivier Mestre,et al.  Calibrated Ensemble Forecasts Using Quantile Regression Forests and Ensemble Model Output Statistics , 2016 .

[15]  Ker-Chau Li,et al.  Sliced Inverse Regression for Dimension Reduction , 1991 .

[16]  Nicolai Meinshausen,et al.  Quantile Regression Forests , 2006, J. Mach. Learn. Res..

[17]  Eibe Frank,et al.  A Simple Approach to Ordinal Classification , 2001, ECML.

[18]  Nikolay Laptev,et al.  Deep and Confident Prediction for Time Series at Uber , 2017, 2017 IEEE International Conference on Data Mining Workshops (ICDMW).

[19]  L. Wasserman.  All of Nonparametric Statistics , 2005 .

[20]  M. Escobar,et al.  Bayesian Density Estimation and Inference Using Mixtures , 1995 .

[21]  Anthony S. Tay,et al.  Evaluating Density Forecasts , 1997 .

[22]  C. Thane,et al.  Conditional Gaussian mixture modelling for dietary pattern analysis , 2007 .

[23]  Rob J Hyndman,et al.  Estimating and Visualizing Conditional Densities , 1996 .

[24]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[25]  Gianluca Pollastri,et al.  A neural network approach to ordinal regression , 2007, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[26]  Francisco C. Pereira,et al.  Beyond Expectation: Deep Joint Mean and Quantile Regression for Spatiotemporal Problems. , 2018, IEEE transactions on neural networks and learning systems.

[27]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[28]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[29]  Liang Peng,et al.  Approximating conditional density functions using dimension reduction , 2009 .

[30]  Alexander G. Gray,et al.  Fast Nonparametric Conditional Density Estimation , 2007, UAI.

[31]  Peter Stone,et al.  Modeling Auction Price Uncertainty Using Boosting-based Conditional Density Estimation , 2002, ICML.

[32]  Rob J Hyndman,et al.  Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond , 2016 .

[33]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[34]  Tom Wilson,et al.  Probabilistic regional population forecasts: The example of Queensland, Australia , 2007 .