Manifold Modeling in Embedded Space: A Perspective for Interpreting "Deep Image Prior"

Deep image prior (DIP), which utilizes a deep convolutional network (ConvNet) structure itself as an image prior, has attractive attentions in computer vision community. It empirically showed that the effectiveness of ConvNet structure in various image restoration applications. However, why the DIP works so well is still in black box, and why ConvNet is essential for images is not very clear. In this study, we tackle this question by considering the convolution divided into "embedding" and "transformation", and proposing a simple, but essential, modeling approach of images/tensors related with dynamical system or self-similarity. The proposed approach named as manifold modeling in embedded space (MMES) can be implemented by using a denoising-auto-encoder in combination with multiway delay-embedding transform. In spite of its simplicity, the image/tensor completion and super-resolution results of MMES were very similar even competitive with DIP in our experiments, and these results would help us for reinterpreting/characterizing the DIP from a perspective of "smooth patch-manifold prior".

[1]  Curtis R. Vogel,et al.  Ieee Transactions on Image Processing Fast, Robust Total Variation{based Reconstruction of Noisy, Blurred Images , 2022 .

[2]  H. Hotelling.  Analysis of a complex of statistical variables into principal components. , 1933 .

[3]  Michal Irani,et al.  InGAN: Capturing and Retargeting the “DNA” of a Natural Image , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[4]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[5]  Ling Shao,et al.  Noisy-As-Clean: Learning Unsupervised Denoising from the Corrupted Image , 2019, ArXiv.

[6]  Eric L. W. Grimson,et al.  From Images to Surfaces: A Computational Study of the Human Early Visual System , 1981 .

[7]  Michal Irani,et al.  "Zero-Shot" Super-Resolution Using Deep Internal Learning , 2017, CVPR.

[8]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[9]  Wensheng Zhang,et al.  The Twist Tensor Nuclear Norm for Video Completion , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[10]  Jaakko Lehtinen,et al.  Noise2Noise: Learning Image Restoration without Clean Data , 2018, ICML.

[11]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[12]  Alexandros G. Dimakis,et al.  Compressed Sensing with Deep Image Prior and Learned Regularization , 2018, ArXiv.

[13]  Navdeep Jaitly,et al.  Adversarial Autoencoders , 2015, ArXiv.

[14]  Antonia Creswell,et al.  Denoising Adversarial Autoencoders , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[15]  James P. Crutchfield,et al.  Geometry from a Time Series , 1980 .

[16]  Martin Szummer,et al.  Temporal texture modeling , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[17]  Huachun Tan,et al.  A Fused CP Factorization Method for Incomplete Tensors , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[18]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[19]  Loïc Royer,et al.  Noise2Self: Blind Denoising by Self-Supervision , 2019, ICML.

[20]  Hidekata Hontani,et al.  Missing Slice Recovery for Tensors Using a Low-Rank Model in Embedded Space , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Karen O. Egiazarian,et al.  Color Image Denoising via Sparse 3D Collaborative Filtering with Grouping Constraint in Luminance-Chrominance Space , 2007, 2007 IEEE International Conference on Image Processing.

[22]  Tao Ding,et al.  A Rank Minimization Approach to Video Inpainting , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[23]  Hidekata Hontani,et al.  Simultaneous Visual Data Completion and Denoising Based on Tensor Rank and Total Variation Minimization and Its Primal-Dual Splitting Algorithm , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  François Malgouyres,et al.  Total variation based interpolation , 1998, 9th European Signal Processing Conference (EUSIPCO 1998).

[25]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[26]  Florian Jug,et al.  Noise2Void - Learning Denoising From Single Noisy Images , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Michal Irani,et al.  “Double-DIP”: Unsupervised Image Decomposition via Coupled Deep-Image-Priors , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Lei Zhang,et al.  Weighted Nuclear Norm Minimization with Application to Image Denoising , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Gabriel Peyré,et al.  Manifold models for signals and images , 2009, Comput. Vis. Image Underst..

[30]  Zuowei Shen,et al.  Robust video denoising using low rank matrix completion , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[31]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[32]  Stan Z. Li,et al.  Markov Random Field Models in Computer Vision , 1994, ECCV.

[33]  Jonathan Cheung-Wai Chan,et al.  Enhanced Sparsity Prior Model for Low-Rank Tensor Completion , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[34]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[35]  Yoshua Bengio,et al.  What regularized auto-encoders learn from the data-generating distribution , 2012, J. Mach. Learn. Res..

[36]  Tomer Michaeli,et al.  Multi-scale Weighted Nuclear Norm Image Restoration , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[37]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[38]  Hidekata Hontani,et al.  Dynamic PET Image Reconstruction Using Nonnegative Matrix Factorization Incorporated With Deep Image Prior , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[39]  Hidekata Hontani,et al.  Simultaneous Tensor Completion and Denoising by Noise Inequality Constrained Convex Optimization , 2018, IEEE Access.

[40]  Joan Bruna,et al.  Deep Geometric Prior for Surface Reconstruction , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Erkki Oja,et al.  Independent Component Analysis , 2001 .

[42]  Michael Elad,et al.  Image Denoising Via Learned Dictionaries and Sparse representation , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[43]  Tali Dekel,et al.  SinGAN: Learning a Generative Model From a Single Natural Image , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[44]  J. Mixter.  Fast , 2012 .

[45]  Ciprian Catana,et al.  PET Image Reconstruction Using Deep Image Prior , 2019, IEEE Transactions on Medical Imaging.

[46]  Angshul Majumdar,et al.  Blind Denoising Autoencoder , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[47]  Xiongjun Zhang,et al.  A Nonconvex Relaxation Approach to Low-Rank Tensor Completion , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[48]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[49]  F. L. Hitchcock.  The Expression of a Tensor or a Polyadic as a Sum of Products , 1927 .

[50]  Andrzej Cichocki,et al.  Smooth PARAFAC Decomposition for Tensor Completion , 2015, IEEE Transactions on Signal Processing.

[51]  Noboru Murata,et al.  Transportation analysis of denoising autoencoders: a novel method for analyzing deep neural networks , 2017, ArXiv.

[52]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[53]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[54]  Reinhard Heckel,et al.  Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks , 2018, ICLR.

[55]  Tomaso Poggio,et al.  Computational vision and regularization theory , 1985, Nature.

[56]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[57]  Taesup Moon,et al.  GAN2GAN: Generative Noise Learning for Blind Image Denoising with Single Noisy Images , 2019, ArXiv.

[58]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[59]  Misha Elena Kilmer,et al.  Novel Methods for Multilinear Data Completion and De-noising Based on Tensor-SVD , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[60]  Ivan Markovsky,et al.  Structured low-rank approximation and its applications , 2008, Autom..

[61]  V. Aggarwal,et al.  Efficient Low Rank Tensor Ring Completion , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[62]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[63]  Jaakko Lehtinen,et al.  High-Quality Self-Supervised Deep Image Denoising , 2019, NeurIPS.

[64]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[65]  P. Van Overschee,et al.  Subspace algorithms for the stochastic identification problem , 1991 .

[66]  Geoffrey Ye Li,et al.  A parameter estimation scheme for damped sinusoidal signals based on low-rank Hankel approximation , 1997, IEEE Trans. Signal Process..

[67]  Wen Gao,et al.  Group-Based Sparse Representation for Image Restoration , 2014, IEEE Transactions on Image Processing.

[68]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[69]  Liqing Zhang,et al.  Bayesian CP Factorization of Incomplete Tensors with Automatic Rank Determination , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[70]  Shakir Mohamed,et al.  Variational Approaches for Auto-Encoding Generative Adversarial Networks , 2017, ArXiv.

[71]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[72]  Qiquan Shi,et al.  Feature Extraction for Incomplete Data Via Low-Rank Tensor Decomposition With Feature Regularization , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[73]  Christopher M. Bishop,et al.  Current address: Microsoft Research, , 2022 .

[74]  B. De Moor,et al.  Subspace algorithms for the stochastic identification problem , 1991, [1991] Proceedings of the 30th IEEE Conference on Decision and Control.

[75]  Wotao Yin,et al.  Parallel matrix factorization for low-rank tensor completion , 2013, ArXiv.

[76]  David Zhang,et al.  Multi-channel Weighted Nuclear Norm Minimization for Real Color Image Denoising , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[77]  Yanning Zhang,et al.  Accurate Tensor Completion via Adaptive Low-Rank Representation , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[78]  Stanley Osher,et al.  Low Dimensional Manifold Model for Image Processing , 2017, SIAM J. Imaging Sci..

[79]  Jieping Ye,et al.  Tensor Completion for Estimating Missing Values in Visual Data , 2013, IEEE Trans. Pattern Anal. Mach. Intell..