Minimizing Trajectory Curvature of ODE-based Generative Models

Recent ODE/SDE-based generative models, such as diffusion models, rectified flows, and flow matching, define a generative process as a time reversal of a fixed forward process. Even though these models show impressive performance on large-scale datasets, numerical simulation requires multiple evaluations of a neural network, leading to a slow sampling speed. We attribute the reason to the high curvature of the learned generative trajectories, as it is directly related to the truncation error of a numerical solver. Based on the relationship between the forward process and the curvature, here we present an efficient method of training the forward process to minimize the curvature of generative trajectories without any ODE/SDE simulation. Experiments show that our method achieves a lower curvature than previous models and, therefore, decreased sampling costs while maintaining competitive performance. Code is available at https://github.com/sangyun884/fast-ode.

[1]  Ricky T. Q. Chen,et al.  Multisample Flow Matching: Straightening Flows with Minibatch Couplings , 2023, ICML.

[2]  Nicholas M. Boffi,et al.  Stochastic Interpolants: A Unifying Framework for Flows and Diffusions , 2023, ArXiv.

[3]  K. Azizzadenesheli,et al.  Fast Sampling of Diffusion Models via Operator Learning , 2022, ICML.

[4]  J. Susskind,et al.  f-DM: A Multi-stage Diffusion Model via Progressive Signal Transformation , 2022, ArXiv.

[5]  Ricky T. Q. Chen,et al.  Flow Matching for Generative Modeling , 2022, ICLR.

[6]  M. S. Albergo,et al.  Building Normalizing Flows with Stochastic Interpolants , 2022, ICLR.

[7]  Qiang Liu Rectified Flow: A Marginal Preserving Approach to Optimal Transport , 2022, ArXiv.

[8]  A. Dimakis,et al.  Soft Diffusion: Score Matching for General Corruptions , 2022, ArXiv.

[9]  E. Hoogeboom,et al.  Blurring Diffusion Models , 2022, ICLR.

[10]  Chengyue Gong,et al.  Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow , 2022, ICLR.

[11]  Furong Huang,et al.  Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise , 2022, NeurIPS.

[12]  Jong-Chul Ye,et al.  Progressive Deblurring of Diffusion Models for Coarse-to-Fine Image Synthesis , 2022, ArXiv.

[13]  A. Solin,et al.  Generative Modelling With Inverse Heat Dissipation , 2022, ICLR.

[14]  Cheng Lu,et al.  DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps , 2022, NeurIPS.

[15]  Tero Karras,et al.  Elucidating the Design Space of Diffusion-Based Generative Models , 2022, NeurIPS.

[16]  David J. Fleet,et al.  Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding , 2022, NeurIPS.

[17]  Yongxin Chen,et al.  Fast Sampling of Diffusion Models with Exponential Integrator , 2022, ICLR.

[18]  Prafulla Dhariwal,et al.  Hierarchical Text-Conditional Image Generation with CLIP Latents , 2022, ArXiv.

[19]  Tim Salimans,et al.  Progressive Distillation for Fast Sampling of Diffusion Models , 2022, ICLR.

[20]  Karsten Kreis,et al.  Tackling the Generative Learning Trilemma with Denoising Diffusion GANs , 2021, ICLR.

[21]  Yongxin Chen,et al.  Diffusion Normalizing Flow , 2021, NeurIPS.

[22]  Diederik P. Kingma,et al.  Variational Diffusion Models , 2021, ArXiv.

[23]  Jan Kautz,et al.  Score-based Generative Modeling in Latent Space , 2021, NeurIPS.

[24]  Prafulla Dhariwal,et al.  Diffusion Models Beat GANs on Image Synthesis , 2021, NeurIPS.

[25]  Eric Luhman,et al.  Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed , 2021, ArXiv.

[26]  Abhishek Kumar,et al.  Score-Based Generative Modeling through Stochastic Differential Equations , 2020, ICLR.

[27]  Rewon Child,et al.  Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images , 2020, ICLR.

[28]  Matthew J. Johnson,et al.  Learning Differential Equations that are Easy to Solve , 2020, NeurIPS.

[29]  Jan Kautz,et al.  NVAE: A Deep Hierarchical Variational Autoencoder , 2020, NeurIPS.

[30]  Pieter Abbeel,et al.  Denoising Diffusion Probabilistic Models , 2020, NeurIPS.

[31]  Song Han,et al.  Differentiable Augmentation for Data-Efficient GAN Training , 2020, NeurIPS.

[32]  Tero Karras,et al.  Training Generative Adversarial Networks with Limited Data , 2020, NeurIPS.

[33]  Adam M. Oberman,et al.  How to train your neural ODE , 2020, ICML.

[34]  Tero Karras,et al.  Analyzing and Improving the Image Quality of StyleGAN , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Johannes L. Schönberger,et al.  SciPy 1.0: fundamental algorithms for scientific computing in Python , 2019, Nature Methods.

[36]  Yang Song,et al.  Generative Modeling by Estimating Gradients of the Data Distribution , 2019, NeurIPS.

[37]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[39]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[40]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[41]  Christopher Burgess,et al.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework , 2016, ICLR 2016.

[42]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[43]  Surya Ganguli,et al.  Deep Unsupervised Learning using Nonequilibrium Thermodynamics , 2015, ICML.

[44]  Aaron C. Courville,et al.  Generative Adversarial Nets , 2014, NIPS.

[45]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[46]  Pascal Vincent,et al.  A Connection Between Score Matching and Denoising Autoencoders , 2011, Neural Computation.

[47]  Yann Brenier,et al.  A computational fluid mechanics solution to the Monge-Kantorovich mass transfer problem , 2000, Numerische Mathematik.