Projecting to Manifolds via Unsupervised Learning

We present a new mechanism, called adversarial projection, that projects a given signal onto the intrinsically low dimensional manifold of true data. This operator can be used for solving inverse problems, which consists of recovering a signal from a collection of noisy measurements. Rather than attempt to encode prior knowledge via an analytic regularizer, we leverage available data to project signals directly onto the (possibly nonlinear) manifold of true data (i.e., regularize via an indicator function of the manifold). Our approach avoids the difficult task of forming a direct representation of the manifold. Instead, we directly learn the projection operator by solving a sequence of unsupervised learning problems, and we prove our method converges in probability to the desired projection. This operator can then be directly incorporated into optimization algorithms in the same manner as Plug-and-Play methods, but now with robust theoretical guarantees. Numerical examples are provided.

[1]  Michael Elad,et al.  Regularization by Denoising via Fixed-Point Projection (RED-PRO) , 2020, SIAM J. Imaging Sci..

[2]  Samy Wu Fung,et al.  OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport , 2020, AAAI.

[3]  Lars Ruthotto,et al.  PNKH-B: A Projected Newton-Krylov Method for Large-Scale Bound-Constrained Optimization , 2020, SIAM J. Sci. Comput..

[4]  Maximilian Schmidt,et al.  Computed tomography reconstruction using deep image prior and learned reconstruction methods , 2020, Inverse Problems.

[5]  Samy Wu Fung,et al.  APAC-Net: Alternating the Population and Agent Control via Two Neural Networks to Solve High-Dimensional Stochastic Mean Field Games , 2020, ArXiv.

[6]  Samy Wu Fung,et al.  A machine learning framework for solving high-dimensional mean field game and mean field control problems , 2019, Proceedings of the National Academy of Sciences.

[7]  Lars Ruthotto,et al.  ADMM-Softmax: an ADMM approach for multinomial logistic regression , 2019, ETNA - Electronic Transactions on Numerical Analysis.

[8]  Samy Wu Fung,et al.  Multigrid Optimization for Large-Scale Ptychographic Phase Retrieval , 2018, SIAM J. Imaging Sci..

[9]  Raymond H. Chan,et al.  A two-stage method for spectral–spatial classification of hyperspectral images , 2018, Journal of Mathematical Imaging and Vision.

[10]  M. Verhaegen,et al.  Phase Retrieval with Sparse Phase Constraint , 2018, SIAM J. Math. Data Sci..

[11]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[12]  Akinori Tanaka,et al.  Discriminator optimal transport , 2019, NeurIPS.

[13]  P. Maass,et al.  The LoDoPaB-CT Dataset: A Benchmark Dataset for Low-Dose CT Reconstruction Methods , 2019, ArXiv.

[14]  Keegan Lensink,et al.  Fluid Flow Mass Transport for Generative Networks , 2019, ArXiv.

[15]  Cem Anil,et al.  Sorting out Lipschitz function approximation , 2018, ICML.

[16]  Lars Ruthotto,et al.  An Uncertainty-Weighted Asynchronous ADMM Method for Parallel PDE Parameter Estimation , 2018, SIAM J. Sci. Comput..

[17]  Eldad Haber,et al.  Deep Neural Networks Motivated by Partial Differential Equations , 2018, Journal of Mathematical Imaging and Vision.

[18]  Lars Ruthotto,et al.  A multiscale method for model order reduction in PDE parameter estimation , 2017, J. Comput. Appl. Math..

[19]  Carola-Bibiane Schönlieb,et al.  Adversarial Regularizers in Inverse Problems , 2018, NeurIPS.

[20]  Chinmay Hegde,et al.  Solving Linear Inverse Problems Using Gan Priors: An Algorithm with Provable Guarantees , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[21]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[22]  Jonas Adler,et al.  Learned Primal-Dual Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[23]  Thomas Pock,et al.  Learning a variational network for reconstruction of accelerated MRI data , 2017, Magnetic resonance in medicine.

[24]  Mihai Postolache,et al.  Strong convergence of Halpern method for firmly type nonexpansive mappings , 2017 .

[25]  Thomas Pock,et al.  Variational Networks: Connecting Variational Methods and Deep Learning , 2017, GCPR.

[26]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[27]  Eldad Haber,et al.  Stable architectures for deep neural networks , 2017, ArXiv.

[28]  Jonas Adler,et al.  Solving ill-posed inverse problems using iterative deep neural networks , 2017, ArXiv.

[29]  Michael Möller,et al.  Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[30]  Carola-Bibiane Schönlieb,et al.  Learning Filter Functions in Regularisers by Minimising Quotients , 2017, SSVM.

[31]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[32]  Chun-Liang Li,et al.  One Network to Solve Them All — Solving Linear Inverse Problems Using Deep Projection Models , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[33]  Feng Lin,et al.  Low-Dose CT With a Residual Encoder-Decoder Convolutional Neural Network , 2017, IEEE Transactions on Medical Imaging.

[34]  Stanley Osher,et al.  Low Dimensional Manifold Model for Image Processing , 2017, SIAM J. Imaging Sci..

[35]  Michael Unser,et al.  Deep Convolutional Neural Network for Inverse Problems in Imaging , 2016, IEEE Transactions on Image Processing.

[36]  Stefano Soatto,et al.  Entropy-SGD: biasing gradient descent into wide valleys , 2016, ICLR.

[37]  Jorge Nocedal,et al.  On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima , 2016, ICLR.

[38]  Stanley H. Chan,et al.  Plug-and-Play ADMM for Image Restoration: Fixed-Point Convergence and Applications , 2016, IEEE Transactions on Computational Imaging.

[39]  Jian Sun,et al.  Deep ADMM-Net for Compressive Sensing MRI , 2016, NIPS.

[40]  Stéphane Mallat,et al.  Inverse Problems with Invariant Multiscale Statistics , 2016, ArXiv.

[41]  Ge Wang,et al.  A Perspective on Deep Imaging , 2016, IEEE Access.

[42]  Karin Schwab,et al.  Best Approximation In Inner Product Spaces , 2016 .

[43]  F. Santambrogio Optimal Transport for Applied Mathematicians: Calculus of Variations, PDEs, and Modeling , 2015 .

[44]  Peter G. Casazza,et al.  Phase retrieval , 2015, SPIE Optical Engineering + Applications.

[45]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[46]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[47]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[48]  Yonina C. Eldar,et al.  Phase Retrieval via Matrix Completion , 2011, SIAM Rev..

[49]  Filippo Santambrogio,et al.  Optimal Transport for Applied Mathematicians , 2015 .

[50]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[51]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[52]  Brendt Wohlberg,et al.  Plug-and-Play priors for model based reconstruction , 2013, 2013 IEEE Global Conference on Signal and Information Processing.

[53]  James Martin,et al.  A Computational Framework for Infinite-Dimensional Bayesian Inverse Problems Part I: The Linearized Case, with Application to Global Seismic Inversion , 2013, SIAM J. Sci. Comput..

[54]  Guy Gilboa,et al.  Expert Regularizers for Task Specific Processing , 2013, SSVM.

[55]  A. Cegielski Iterative Methods for Fixed Point Problems in Hilbert Spaces , 2012 .

[56]  Lei Zhang,et al.  Low-Dose X-ray CT Reconstruction via Dictionary Learning , 2012, IEEE Transactions on Medical Imaging.

[57]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[58]  Léon Bottou,et al.  Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[59]  S. Arridge,et al.  Optical tomography: forward and inverse problems , 2009, 0907.2586.

[60]  Gabriel Peyré,et al.  Manifold models for signals and images , 2009, Comput. Vis. Image Underst..

[61]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[62]  C. Villani Optimal Transport: Old and New , 2008 .

[63]  Gabriel Peyré,et al.  Image Processing with Nonlocal Spectral Bases , 2008, Multiscale Model. Simul..

[64]  Ameet Talwalkar,et al.  Large-scale manifold learning , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[65]  Hongbin Zha,et al.  Riemannian Manifold Learning , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[66]  Wotao Yin,et al.  Bregman Iterative Algorithms for (cid:2) 1 -Minimization with Applications to Compressed Sensing ∗ , 2008 .

[67]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[68]  H. Robbins A Stochastic Approximation Method , 1951 .

[69]  Jianhong Shen,et al.  Deblurring images: Matrices, spectra, and filtering , 2007, Math. Comput..

[70]  Vin de Silva,et al.  On the Local Behavior of Spaces of Natural Images , 2007, International Journal of Computer Vision.

[71]  Emmanuel J. Candès,et al.  Quantitative Robust Uncertainty Principles and Optimally Sparse Decompositions , 2004, Found. Comput. Math..

[72]  E. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[73]  Lorenzo Rosasco,et al.  Learning from Examples as an Inverse Problem , 2005, J. Mach. Learn. Res..

[74]  Wotao Yin,et al.  An Iterative Regularization Method for Total Variation-Based Image Restoration , 2005, Multiscale Model. Simul..

[75]  E. Haber,et al.  Inversion of 3D electromagnetic data in frequency and time domain using an inexact all-at-once approach , 2004 .

[76]  Kim Steenstrup Pedersen,et al.  The Nonlinear Statistics of High-Contrast Patches in Natural Images , 2003, International Journal of Computer Vision.

[77]  A. Galántai Projectors and Projection Methods , 2003 .

[78]  D. Calvetti,et al.  Tikhonov Regularization of Large Linear Problems , 2003 .

[79]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[80]  Felipe Cucker,et al.  Best Choices for Regularization Parameters in Learning Theory: On the Bias—Variance Problem , 2002, Found. Comput. Math..

[81]  Hong-Kun Xu Iterative Algorithms for Nonlinear Operators , 2002 .

[82]  øöö Blockinø Phase retrieval, error reduction algorithm, and Fienup variants: A view from convex optimization , 2002 .

[83]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[84]  E. Haber,et al.  Fast Simulation of 3D Electromagnetic Problems Using Potentials , 2000 .

[85]  Gene H. Golub,et al.  Tikhonov Regularization and Total Least Squares , 1999, SIAM J. Matrix Anal. Appl..

[86]  S. Arridge Optical tomography in medical imaging , 1999 .

[87]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[88]  Simeon Reich,et al.  CONSTRUCTIVE TECHNIQUES FOR ACCRETIVE AND MONOTONE OPERATORS , 1979 .

[89]  B. Halpern Fixed points of nonexpanding maps , 1967 .