Improving 3D Imaging with Pre-Trained Perpendicular 2D Diffusion Models

Diffusion models have become a popular approach for image generation and reconstruction due to their numerous advantages. However, most diffusion-based inverse problem-solving methods only deal with 2D images, and even recently published 3D methods do not fully exploit the 3D distribution prior. To address this, we propose a novel approach using two perpendicular pre-trained 2D diffusion models to solve the 3D inverse problem. By modeling the 3D data distribution as a product of 2D distributions sliced in different directions, our method effectively addresses the curse of dimensionality. Our experimental results demonstrate that our method is highly effective for 3D medical image reconstruction tasks, including MRI Z-axis super-resolution, compressed sensing MRI, and sparse-view CT. Our method can generate high-quality voxel volumes suitable for medical applications.

[1]  Y. Matias,et al.  Dreamix: Video Diffusion Models are General Video Editors , 2023, ArXiv.

[2]  Jeffrey A. Fessler,et al.  Sparse-View Cone Beam CT Reconstruction Using Data-Consistent Supervised and Adversarial Learning From Scarce Training Data , 2022, IEEE Transactions on Computational Imaging.

[3]  Michael T. McCann,et al.  Solving 3D Inverse Problems Using Pre-Trained 2D Diffusion Models , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  J. C. Ye,et al.  Parallel Diffusion Models of Operator and Image for Blind Inverse Problems , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Michael T. McCann,et al.  Diffusion Posterior Sampling for General Noisy Inverse Problems , 2022, ICLR.

[6]  V. Cevher,et al.  DiGress: Discrete Denoising diffusion for graph generation , 2022, ICLR.

[7]  Jonathan Ho.  Classifier-Free Diffusion Guidance , 2022, ArXiv.

[8]  J. C. Ye,et al.  Improving Diffusion Models for Inverse Problems using Manifold Constraints , 2022, Neural Information Processing Systems.

[9]  Xiang Lisa Li,et al.  Diffusion-LM Improves Controllable Text Generation , 2022, NeurIPS.

[10]  Timothy S. Coalson,et al.  Surface Vision Transformers: Attention-Based Modelling applied to Cortical Analysis , 2022, MIDL.

[11]  Michael Elad,et al.  Denoising Diffusion Restoration Models , 2022, NeurIPS.

[12]  B. Ommer,et al.  High-Resolution Image Synthesis with Latent Diffusion Models , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  S. Ermon,et al.  Solving Inverse Problems in Medical Imaging with Score-Based Generative Models , 2021, ICLR.

[14]  Jong-Chul Ye,et al.  Score-based diffusion models for accelerated MRI , 2021, Medical Image Anal..

[15]  Tasnima Sadekova,et al.  Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech , 2021, ICML.

[16]  Prafulla Dhariwal,et al.  Diffusion Models Beat GANs on Image Synthesis , 2021, NeurIPS.

[17]  Abhishek Kumar,et al.  Score-Based Generative Modeling through Stochastic Differential Equations , 2020, ICLR.

[18]  Pieter Abbeel,et al.  Denoising Diffusion Probabilistic Models , 2020, NeurIPS.

[19]  Bo Zhou,et al.  DuDoRNet: Learning a Dual-Domain Recurrent Network for Fast MRI Reconstruction With Deep T1 Prior , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Yang Song,et al.  Generative Modeling by Estimating Gradients of the Data Distribution , 2019, NeurIPS.

[21]  Baiyu Chen,et al.  Low‐dose CT for the detection and classification of metastatic liver lesions: Results of the 2016 Low Dose CT Grand Challenge , 2017, Medical physics.

[22]  Michael Unser,et al.  Deep Convolutional Neural Network for Inverse Problems in Imaging , 2016, IEEE Transactions on Image Processing.

[23]  Nick C Fox,et al.  Using visual rating to diagnose dementia: a critical evaluation of MRI atrophy scales , 2015, Journal of Neurology, Neurosurgery & Psychiatry.

[24]  Surya Ganguli,et al.  Deep Unsupervised Learning using Nonequilibrium Thermodynamics , 2015, ICML.

[25]  Arno Klein,et al.  Large-scale evaluation of ANTs and FreeSurfer cortical thickness measurements , 2014, NeuroImage.

[26]  B. Efron.  Tweedie’s Formula and Selection Bias , 2011, Journal of the American Statistical Association.

[27]  Pascal Vincent,et al.  A Connection Between Score Matching and Denoising Autoencoders , 2011, Neural Computation.

[28]  D. Donoho,et al.  Sparse MRI: The application of compressed sensing for rapid MR imaging , 2007, Magnetic resonance in medicine.

[29]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[30]  Frederik Barkhof,et al.  White Matter Changes on CT and MRI: An Overview of Visual Rating Scales , 1998, European Neurology.

[31]  B. Anderson.  Reverse-time diffusion equation models , 1982 .