From TV-L1 to Gated Recurrent Nets

TV-L1 is a classical diffusion-reaction model for low-level vision tasks, which can be solved by a duality based iterative algorithm. Considering the recent success of end-to-end learned representations, we propose a TV-LSTM network to unfold the duality based iterations into long short-term memory (LSTM) cells. To provide a trainable network, we relax the difference operators in the gate and cell update of TV-LSTM to trainable parameters. Then, the proposed end-to-end trainable TV-LSTMs can be naturally connected with various task-specific networks, e.g., optical flow estimation and image decomposition. Extensive experiments on optical flow estimation and structure + texture decomposition have demonstrated the effectiveness and efficiency of the proposed method.

[1]  David P. Wipf,et al.  From Bayesian Sparsity to Gated Recurrent Nets , 2017, NIPS.

[2]  Antonin Chambolle,et al.  An Upwind Finite-Difference Method for Total Variation-Based Image Smoothing , 2011, SIAM J. Imaging Sci..

[3]  Javier Sánchez Pérez,et al.  TV-L1 Optical Flow Estimation , 2013, Image Process. Line.

[4]  Bin Dong,et al.  PDE-Net: Learning PDEs from Data , 2017, ICML.

[5]  Yann LeCun,et al.  Learning Fast Approximations of Sparse Coding , 2010, ICML.

[6]  Jose Luis Lisani,et al.  Directional Filters for Cartoon + Texture Image Decomposition , 2016, Image Process. Line.

[7]  Jian Sun,et al.  Deep ADMM-Net for Compressive Sensing MRI , 2016, NIPS.

[8]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[9]  Li Xu,et al.  Structure extraction from texture via relative total variation , 2012, ACM Trans. Graph..

[10]  Yunjin Chen,et al.  Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[12]  Thomas Pock,et al.  Variational Networks: Connecting Variational Methods and Deep Learning , 2017, GCPR.

[13]  Gordon Wetzstein,et al.  Unrolled Optimization with Deep Priors , 2017, ArXiv.

[14]  Thomas Pock,et al.  A Primal Dual Network for Low-Level Vision Problems , 2017, GCPR.

[15]  ANTONIN CHAMBOLLE,et al.  An Algorithm for Total Variation Minimization and Applications , 2004, Journal of Mathematical Imaging and Vision.

[16]  Chuang Gan,et al.  End-to-End Learning of Motion Representation for Video Understanding , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.