Provably Convergent Schrödinger Bridge with Applications to Probabilistic Time Series Imputation

The Schr\"odinger bridge problem (SBP) is gaining increasing attention in generative modeling and showing promising potential even in comparison with the score-based generative models (SGMs). SBP can be interpreted as an entropy-regularized optimal transport problem, which conducts projections onto every other marginal alternatingly. However, in practice, only approximated projections are accessible and their convergence is not well understood. To fill this gap, we present a first convergence analysis of the Schr\"odinger bridge algorithm based on approximated projections. As for its practical applications, we apply SBP to probabilistic time series imputation by generating missing values conditioned on observed data. We show that optimizing the transport cost improves the performance and the proposed algorithm achieves the state-of-the-art result in healthcare and environmental data while exhibiting the advantage of exploring both temporal and feature patterns in probabilistic time series imputation.

[1]  Alain Durmus,et al.  Non-asymptotic convergence bounds for Sinkhorn iterates and their gradients: a coupling approach , 2023, COLT.

[2]  Alain Durmus,et al.  Quantitative contraction rates for Sinkhorn algorithm: beyond bounded costs and compact marginals , 2023, 2304.04451.

[3]  Promit Ghosal,et al.  On the Convergence Rate of Sinkhorn's Algorithm , 2022, 2212.06000.

[4]  Ricky T. Q. Chen,et al.  Neural Conservation Laws: A Divergence-Free Perspective , 2022, NeurIPS.

[5]  Andrej Risteski,et al.  Statistical Efficiency of Score Matching: The View from Isoperimetry , 2022, ICLR.

[6]  Anru R. Zhang,et al.  Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions , 2022, ICLR.

[7]  Guang Lin,et al.  An adaptively weighted stochastic gradient MCMC algorithm for Monte Carlo simulation and global optimization , 2022, Statistics and Computing.

[8]  Holden Lee,et al.  Convergence for score-based generative modeling with polynomial complexity , 2022, NeurIPS.

[9]  F. Santambrogio,et al.  The flow map of the Fokker-Planck equation does not provide optimal transport , 2022, Appl. Math. Lett..

[10]  Valentin De Bortoli,et al.  Conditional Simulation Using Diffusion Schrödinger Bridges , 2022, UAI.

[11]  I. Oseledets,et al.  Understanding DDPM Latent Codes Through Optimal Transport , 2022, ICLR.

[12]  Marco Cuturi,et al.  The Schrödinger Bridge between Gaussian Measures has a Closed Form , 2022, AISTATS.

[13]  Marcel Nutz,et al.  Stability of Schrödinger potentials and convergence of Sinkhorn’s algorithm , 2022, The Annals of Probability.

[14]  Evangelos A. Theodorou,et al.  Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory , 2021, ICLR.

[15]  Stefano Ermon,et al.  CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation , 2021, NeurIPS.

[16]  Greg Mori,et al.  Continuous Latent Process Flows , 2021, NeurIPS.

[17]  Gefei Wang,et al.  Deep Generative Learning via Schrödinger Bridge , 2021, ICML.

[18]  Espen Bernton,et al.  Stability of entropic optimal transport and Schrödinger bridges , 2021, Journal of Functional Analysis.

[19]  Neil D. Lawrence,et al.  Solving Schrödinger Bridges via Maximum Likelihood , 2021, Entropy.

[20]  Valentin De Bortoli,et al.  Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling , 2021, NeurIPS.

[21]  Tryphon T. Georgiou,et al.  Optimal Transport in Systems and Control , 2021, Annu. Rev. Control. Robotics Auton. Syst..

[22]  Roland Vollgraf,et al.  Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting , 2021, ICML.

[23]  Iain Murray,et al.  Maximum Likelihood Training of Score-Based Diffusion Models , 2021, NeurIPS.

[24]  Abhishek Kumar,et al.  Score-Based Generative Modeling through Stochastic Differential Equations , 2020, ICLR.

[25]  F. Liang,et al.  A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributions , 2020, NeurIPS.

[26]  Pieter Abbeel,et al.  Denoising Diffusion Probabilistic Models , 2020, NeurIPS.

[27]  Satya Narayan Shukla,et al.  Multi-Time Attention Networks for Irregularly Sampled Time Series , 2020, ICLR.

[28]  Terry Lyons,et al.  Neural Controlled Differential Equations for Irregular Time Series , 2020, NeurIPS.

[29]  David Duvenaud,et al.  Scalable Gradients for Stochastic Differential Equations , 2020, AISTATS.

[30]  Kenneth F. Caluya,et al.  Wasserstein Proximal Algorithms for the Schrödinger Bridge Problem: Density Control With Nonlinear Drift , 2019, IEEE Transactions on Automatic Control.

[31]  Ruoxuan Xiong,et al.  Large Dimensional Latent Factor Modeling with Missing Observations and Applications to Causal Inference , 2019, SSRN Electronic Journal.

[32]  Michael Bohlke-Schneider,et al.  High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes , 2019, NeurIPS.

[33]  Heung-Il Suk,et al.  Uncertainty-Aware Variational-Recurrent Imputation Network for Clinical Time Series , 2019, IEEE Transactions on Cybernetics.

[34]  Xiaojie Yuan,et al.  E²GAN: End-to-End Generative Adversarial Network for Multivariate Time Series Imputation , 2019, IJCAI.

[35]  Yang Song,et al.  Generative Modeling by Estimating Gradients of the Data Distribution , 2019, NeurIPS.

[36]  David Duvenaud,et al.  Latent ODEs for Irregularly-Sampled Time Series , 2019, ArXiv.

[37]  Gunnar Rätsch,et al.  GP-VAE: Deep Probabilistic Time Series Imputation , 2019, AISTATS.

[38]  David Duvenaud,et al.  FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models , 2018, ICLR.

[39]  Michele Pavon,et al.  The Data‐Driven Schrödinger Bridge , 2018, Communications on Pure and Applied Mathematics.

[40]  Lei Li,et al.  BRITS: Bidirectional Recurrent Imputation for Time Series , 2018, NeurIPS.

[41]  Peter F. Christoffersen,et al.  Illiquidity Premia in the Equity Options Market , 2017 .

[42]  Guokun Lai,et al.  Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks , 2017, SIGIR.

[43]  Matus Telgarsky,et al.  Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis , 2017, COLT.

[44]  Yan Liu,et al.  Recurrent Neural Networks for Multivariate Time Series with Missing Values , 2016, Scientific Reports.

[45]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[46]  Surya Ganguli,et al.  Deep Unsupervised Learning using Nonequilibrium Thermodynamics , 2015, ICML.

[47]  Tryphon T. Georgiou,et al.  Stochastic Bridges of Linear Systems , 2014, IEEE Transactions on Automatic Control.

[48]  Yu Zheng,et al.  U-Air: when urban air quality inference meets big data , 2013, KDD.

[49]  Christian L'eonard.  Some properties of path measures , 2013, 1308.0217.

[50]  Christian L'eonard.  A survey of the Schr\"odinger problem and some of its connections with optimal transport , 2013, 1308.0215.

[51]  G. Moody,et al.  Predicting in-hospital mortality of ICU patients: The PhysioNet/Computing in cardiology challenge 2012 , 2012, 2012 Computing in Cardiology.

[52]  J. Ma,et al.  Forward-Backward Stochastic Differential Equations and their Applications , 2007 .

[53]  L. Rüschendorf.  Convergence of the iterative proportional fitting procedure , 1995 .

[54]  R Malladi,et al.  Image processing via level set curvature flow. , 1995, Proceedings of the National Academy of Sciences of the United States of America.

[55]  B. Anderson.  Reverse-time diffusion equation models , 1982 .

[56]  S. Kullback.  Probability Densities with Given Marginals , 1968 .

[57]  Marcel Nutz.  Introduction to Entropic Optimal Transport , 2021 .

[58]  T. Georgiou,et al.  Stochastic Control Liaisons: Richard Sinkhorn Meets Gaspard Monge on a Schrödinger Bridge , 2021, SIAM Rev..

[59]  Patrick Gallinari,et al.  Normalizing Kalman Filters for Multivariate Time Series Analysis , 2020, NeurIPS.

[60]  David A. Clifton,et al.  Multitask Gaussian Processes for Multivariate Physiological Time-Series Analysis , 2015, IEEE Transactions on Biomedical Engineering.

[61]  M. Hutchinson.  A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines , 1989 .