DiffSTG: Probabilistic Spatio-Temporal Graph Forecasting with Denoising Diffusion Models

Spatio-temporal graph neural networks (STGNN) have emerged as the dominant model for spatio-temporal graph (STG) forecasting. Despite their success, they fail to model intrinsic uncertainties within STG data, which cripples their practicality in downstream tasks for decision-making. To this end, this paper focuses on probabilistic STG forecasting, which is challenging due to the difficulty in modeling uncertainties and complex ST dependencies. In this study, we present the first attempt to generalize the popular de-noising diffusion probabilistic models to STGs, leading to a novel non-autoregressive framework called DiffSTG, along with the first denoising network UGnet for STG in the framework. Our approach combines the spatio-temporal learning capabilities of STGNNs with the uncertainty measurements of diffusion models. Extensive experiments validate that DiffSTG reduces the Continuous Ranked Probability Score (CRPS) by 4%-14%, and Root Mean Squared Error (RMSE) by 2%-7% over existing methods on three real-world datasets.

[1]  Haomin Wen,et al.  Spatial-Temporal Position-Aware Graph Convolution Networks for Traffic Flow Forecasting , 2023, IEEE Transactions on Intelligent Transportation Systems.

[2]  Huan Yee Koh,et al.  A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection , 2023, ArXiv.

[3]  Yang Wang,et al.  DiffLoad: Uncertainty Quantification in Load Forecasting with Diffusion Model , 2023, ArXiv.

[4]  S. Quan,et al.  A transformer-based diffusion probabilistic model for heart rate and blood pressure forecasting in Intensive Care Unit , 2023, Comput. Methods Programs Biomed..

[5]  Xin-xin Lu,et al.  Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement , 2023, NeurIPS.

[6]  Jonathan Dumas,et al.  Denoising Diffusion Probabilistic Models for Probabilistic Energy Forecasting , 2022, 2023 IEEE Belgrade PowerTech.

[7]  Yiwei Wang,et al.  AirFormer: Predicting Nationwide Air Quality in China with Transformers , 2022, AAAI.

[8]  Haomin Wen,et al.  Traffic Inflow and Outflow Forecasting by Modeling Intra- and Inter-Relationship Between Flows , 2022, IEEE Transactions on Intelligent Transportation Systems.

[9]  Shuo Shang,et al.  MSDR: Multi-Step Dependency Relation Networks for Spatial Temporal Forecasting , 2022, KDD.

[10]  Tian Zhou,et al.  Learning to Rotate: Quaternion Transformer for Complicated Periodical Time Series Forecasting , 2022, KDD.

[11]  Jingyuan Wang,et al.  STDEN: Towards Physics-Guided Neural Networks for Traffic Flow Prediction , 2022, AAAI.

[12]  Vikram S. Voleti,et al.  MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation , 2022, ArXiv.

[13]  Junchi Yan,et al.  Transformers in Time Series: A Survey , 2022, IJCAI.

[14]  Tian Zhou,et al.  FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting , 2022, ICML.

[15]  B. Ommer,et al.  High-Resolution Image Synthesis with Latent Diffusion Models , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Noseong Park,et al.  Graph Neural Controlled Differential Equations for Traffic Forecasting , 2021, AAAI.

[17]  Rafael E. Carrillo,et al.  Spatio-Temporal Graph Neural Networks for Multi-Site PV Power Forecasting , 2021, IEEE Transactions on Sustainable Energy.

[18]  Stefano Ermon,et al.  CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation , 2021, NeurIPS.

[19]  Mark Coates,et al.  RNN with Particle Flow for Probabilistic Spatio-temporal Forecasting , 2021, ICML.

[20]  Rose Yu,et al.  Quantifying Uncertainty in Deep Spatiotemporal Forecasting , 2021, KDD.

[21]  Zhou Zhao,et al.  DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism , 2021, AAAI.

[22]  Gao Cong,et al.  Learning Dynamics and Heterogeneity of Spatial-Temporal Graph Data for Traffic Forecasting , 2021, IEEE Transactions on Knowledge and Data Engineering.

[23]  Roland Vollgraf,et al.  Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting , 2021, ICML.

[24]  Hui Xiong,et al.  Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting , 2020, AAAI.

[25]  Jiaming Song,et al.  Denoising Diffusion Implicit Models , 2020, ICLR.

[26]  Bryan Catanzaro,et al.  DiffWave: A Versatile Diffusion Model for Audio Synthesis , 2020, ICLR.

[27]  Heiga Zen,et al.  WaveGrad: Estimating Gradients for Waveform Generation , 2020, ICLR.

[28]  Pieter Abbeel,et al.  Denoising Diffusion Probabilistic Models , 2020, NeurIPS.

[29]  Stefano Ermon,et al.  Improved Techniques for Training Score-Based Generative Models , 2020, NeurIPS.

[30]  Sungwon Kim,et al.  Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search , 2020, NeurIPS.

[31]  Youfang Lin,et al.  Spatial-Temporal Synchronous Graph Convolutional Networks: A New Framework for Spatial-Temporal Network Data Forecasting , 2020, AAAI.

[32]  Stefano Ermon,et al.  Permutation Invariant Graph Generation via Score-Based Generative Modeling , 2020, AISTATS.

[33]  Philip S. Yu,et al.  Spatial temporal incidence dynamic graph neural networks for traffic flow forecasting , 2020, Inf. Sci..

[34]  Yang Song,et al.  Generative Modeling by Estimating Gradients of the Data Distribution , 2019, NeurIPS.

[35]  Jing Jiang,et al.  Graph WaveNet for Deep Spatial-Temporal Graph Modeling , 2019, IJCAI.

[36]  Zhiyuan Liu,et al.  Graph Neural Networks: A Review of Methods and Applications , 2018, AI Open.

[37]  Yu Zheng,et al.  Deep Distributed Fusion Network for Air Quality Prediction , 2018, KDD.

[38]  Vladlen Koltun,et al.  An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling , 2018, ArXiv.

[39]  Jieping Ye,et al.  Deep Multi-View Spatial-Temporal Network for Taxi Demand Prediction , 2018, AAAI.

[40]  Haoteng Yin,et al.  Spatio-temporal Graph Convolutional Neural Network: A Deep Learning Framework for Traffic Forecasting , 2017, IJCAI.

[41]  Cyrus Shahabi,et al.  Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting , 2017, ICLR.

[42]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[43]  Alex Kendall,et al.  Concrete Dropout , 2017, NIPS.

[44]  Valentin Flunkert,et al.  DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks , 2017, International Journal of Forecasting.

[45]  Heiga Zen,et al.  WaveNet: A Generative Model for Raw Audio , 2016, SSW.

[46]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[47]  Yan Liu,et al.  Recurrent Neural Networks for Multivariate Time Series with Missing Values , 2016, Scientific Reports.

[48]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[49]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[50]  Pascal Vincent,et al.  A Connection Between Score Matching and Denoising Autoencoders , 2011, Neural Computation.

[51]  Aapo Hyvärinen,et al.  Estimation of Non-Normalized Statistical Models by Score Matching , 2005, J. Mach. Learn. Res..

[52]  Lixin Li,et al.  Interpolation methods for spatio-temporal geographic data , 2004, Comput. Environ. Urban Syst..

[53]  R. L. Winkler,et al.  Scoring Rules for Continuous Probability Distributions , 1976 .

[54]  Michael I. Jordan,et al.  AUTO-ENCODING VARIATIONAL BAYES , 2020 .

[55]  David Duvenaud,et al.  Latent Ordinary Differential Equations for Irregularly-Sampled Time Series , 2019, NeurIPS.