CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation

The imputation of missing values in time series has many applications in healthcare and finance. While autoregressive models are natural candidates for time series imputation, score-based diffusion models have recently outperformed existing counterparts including autoregressive models in many tasks such as image generation and audio synthesis, and would be promising for time series imputation. In this paper, we propose Conditional Score-based Diffusion models for Imputation (CSDI), a novel time series imputation method that utilizes score-based diffusion models conditioned on observed data. Unlike existing score-based approaches, the conditional diffusion model is explicitly trained for imputation and can exploit correlations between observed values. On healthcare and environmental data, CSDI improves by 40-65% over existing probabilistic imputation methods on popular performance metrics. In addition, deterministic imputation by CSDI reduces the error by 5-20% compared to the state-of-the-art deterministic imputation methods. Furthermore, CSDI can also be applied to time series interpolation and probabilistic forecasting, and is competitive with existing baselines. The code is available at https://github.com/ermongroup/CSDI.

[1]  Jinsung Yoon,et al.  Estimating Missing Data in Temporal Data Streams Using Multi-Directional Recurrent Neural Networks , 2017, IEEE Transactions on Biomedical Engineering.

[2]  Satya Narayan Shukla,et al.  Multi-Time Attention Networks for Irregularly Sampled Time Series , 2020, ICLR.

[3]  Andrew Gordon Wilson,et al.  GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration , 2018, NeurIPS.

[4]  Xiaojie Yuan,et al.  E²GAN: End-to-End Generative Adversarial Network for Multivariate Time Series Imputation , 2019, IJCAI.

[5]  Wei Ping,et al.  DiffWave: A Versatile Diffusion Model for Audio Synthesis , 2020, ICLR.

[6]  Jong-Hwan Kim,et al.  RDIS: Random Drop Imputation with Self-Training for Incomplete Time Series Data , 2020, ArXiv.

[7]  Surya Ganguli,et al.  Deep Unsupervised Learning using Nonequilibrium Thermodynamics , 2015, ICML.

[8]  Wei Cao,et al.  BRITS: Bidirectional Recurrent Imputation for Time Series , 2018, NeurIPS.

[9]  Guangdong Feng,et al.  A Tensor Based Method for Missing Traffic Data Completion , 2013 .

[10]  Zahra Kadkhodaie,et al.  Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser , 2020, ArXiv.

[11]  Hongyuan Zha,et al.  Transformer Hawkes Process , 2020, ICML.

[12]  Michael Bohlke-Schneider,et al.  High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes , 2019, NeurIPS.

[13]  Nam Nguyen,et al.  Temporal Latent Auto-Encoder: A Method for Probabilistic Multivariate Time Series Forecasting , 2021, AAAI.

[14]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[15]  G. Moody,et al.  Predicting in-hospital mortality of ICU patients: The PhysioNet/Computing in cardiology challenge 2012 , 2012, 2012 Computing in Cardiology.

[16]  Yan Liu,et al.  Recurrent Neural Networks for Multivariate Time Series with Missing Values , 2016, Scientific Reports.

[17]  Syama Sundar Rangapuram,et al.  GluonTS: Probabilistic and Neural Time Series Modeling in Python , 2020, J. Mach. Learn. Res..

[18]  Edwin V. Bonilla,et al.  Multi-task Gaussian Process Prediction , 2007, NIPS.

[19]  Ingmar Schuster,et al.  Multi-variate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows , 2020, ICLR.

[20]  Stefano Ermon,et al.  Improved Techniques for Training Score-Based Generative Models , 2020, NeurIPS.

[21]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[22]  Yang Song,et al.  Generative Modeling by Estimating Gradients of the Data Distribution , 2019, NeurIPS.

[23]  R. L. Winkler,et al.  Scoring Rules for Continuous Probability Distributions , 1976 .

[24]  Heiga Zen,et al.  WaveGrad: Estimating Gradients for Waveform Generation , 2021, ICLR.

[25]  Ying Zhang,et al.  Multivariate Time Series Imputation with Generative Adversarial Networks , 2018, NeurIPS.

[26]  Shuai Yi,et al.  Efficient Attention: Attention with Linear Complexities , 2018, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[27]  Pieter Abbeel,et al.  Denoising Diffusion Probabilistic Models , 2020, NeurIPS.

[28]  Guokun Lai,et al.  Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks , 2017, SIGIR.

[29]  Stef van Buuren,et al.  MICE: Multivariate Imputation by Chained Equations in R , 2011 .

[30]  Stephan Mandt,et al.  GP-VAE: Deep Probabilistic Time Series Imputation , 2020, AISTATS.

[31]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[32]  Curtis Hawthorne,et al.  Symbolic Music Generation with Diffusion Models , 2021, ISMIR.

[33]  Heung-Il Suk,et al.  Uncertainty-Aware Variational-Recurrent Imputation Network for Clinical Time Series , 2019, IEEE Transactions on Cybernetics.

[34]  Stefano Ermon,et al.  Permutation Invariant Graph Generation via Score-Based Generative Modeling , 2020, AISTATS.

[35]  Tshilidzi Marwala,et al.  Missing data: A comparison of neural network and expectation maximization techniques , 2007 .

[36]  Prafulla Dhariwal,et al.  Improved Denoising Diffusion Probabilistic Models , 2021, ICML.

[37]  Abhishek Kumar,et al.  Score-Based Generative Modeling through Stochastic Differential Equations , 2020, ICLR.

[38]  Jiaming Song,et al.  Denoising Diffusion Implicit Models , 2021, ICLR.

[39]  David Duvenaud,et al.  Latent Ordinary Differential Equations for Irregularly-Sampled Time Series , 2019, NeurIPS.

[40]  Jianwei Yin,et al.  Generative Semi-supervised Learning for Multivariate Time Series Imputation , 2021, AAAI.

[41]  Changyou Chen,et al.  GLIMA: Global and Local Time Series Imputation with Multi-directional Attention Learning , 2020, 2020 IEEE International Conference on Big Data (Big Data).

[42]  A. Hudak,et al.  Nearest neighbor imputation of species-level, plot-scale forest structure attributes from LiDAR data , 2008 .

[43]  Tianrui Li,et al.  ST-MVL: Filling Missing Values in Geo-Sensory Time Series Data , 2016, IJCAI.

[44]  Ingmar Schuster,et al.  Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting , 2021, ICML.