Trainable TV-L1 model as recurrent nets for low-level vision

TV- $$L^1$$ is a classical diffusion–reaction model for low-level vision tasks, which can be solved by a duality-based iterative algorithm. Considering the recent success of end-to-end learned representations, we propose a TV-LSTM network to unfold the duality-based iterations of TV- $$L^1$$ into long short-term memory (LSTM) cells. In particular, we formulate the iterations as customized layers of a LSTM neural network. Then, the proposed end-to-end trainable TV-LSTMs can be naturally connected with various task-specific networks, e.g., optical flow, image decomposition and event-based optical flow estimation. Extensive experiments on optical flow estimation and structure + texture decomposition have demonstrated the effectiveness and efficiency of the proposed method.

[1]  Gordon Wetzstein,et al.  Unrolled Optimization with Deep Priors , 2017, ArXiv.

[2]  F. Y. Wu.  The Potts model , 1982 .

[3]  Tobi Delbruck,et al.  A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor , 2014, IEEE Journal of Solid-State Circuits.

[4]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[5]  Vincent Le Guen,et al.  Cartoon + Texture Image Decomposition by the TV-L1 Model , 2014, Image Process. Line.

[6]  M. Nikolova.  An Algorithm for Total Variation Minimization and Applications , 2004 .

[7]  Vladlen Koltun,et al.  High Speed and High Dynamic Range Video with an Event Camera , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Jose Luis Lisani,et al.  Directional Filters for Cartoon + Texture Image Decomposition , 2016, Image Process. Line.

[9]  Jian Sun,et al.  Deep ADMM-Net for Compressive Sensing MRI , 2016, NIPS.

[10]  Horst Bischof,et al.  A Duality Based Approach for Realtime TV-L1 Optical Flow , 2007, DAGM-Symposium.

[11]  Weiwei Sun,et al.  Linearized Multi-Sampling for Differentiable Image Transformation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[12]  Stan Z. Li,et al.  Markov Random Field Modeling in Image Analysis , 2001, Computer Science Workbench.

[13]  David P. Wipf,et al.  From Bayesian Sparsity to Gated Recurrent Nets , 2017, NIPS.

[14]  Chuang Gan,et al.  End-to-End Learning of Motion Representation for Video Understanding , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[15]  Rui Hu,et al.  DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[16]  Li Xu,et al.  Structure extraction from texture via relative total variation , 2012, ACM Trans. Graph..

[17]  Stephen P. Boyd,et al.  End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging , 2018, ACM Trans. Graph..

[18]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[19]  Yann LeCun,et al.  Learning Fast Approximations of Sparse Coding , 2010, ICML.

[20]  Matthias Nießner,et al.  DDSL: Deep Differentiable Simplex Layer for Learning Geometric Signals , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[21]  Chiara Bartolozzi,et al.  Event-Based Vision: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Yunjin Chen,et al.  Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Laurent Condat,et al.  Discrete Total Variation: New Definition and Minimization , 2017, SIAM J. Imaging Sci..

[24]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[25]  Thomas Pock,et al.  Variational Networks: Connecting Variational Methods and Deep Learning , 2017, GCPR.

[26]  Bin Dong,et al.  PDE-Net: Learning PDEs from Data , 2017, ICML.

[27]  Antonin Chambolle,et al.  An Upwind Finite-Difference Method for Total Variation-Based Image Smoothing , 2011, SIAM J. Imaging Sci..

[28]  Lin Sun,et al.  From TV-L1 to Gated Recurrent Nets , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[29]  Javier Sánchez Pérez,et al.  TV-L1 Optical Flow Estimation , 2013, Image Process. Line.