Interpretable Deep Multimodal Image Super-Resolution

Multimodal image super-resolution (SR) is the reconstruction of a high resolution image given a low-resolution observation with the aid of another image modality. While existing deep multimodal models do not incorporate domain knowledge about image SR, we present a multimodal deep network design that integrates coupled sparse priors and allows the effective fusion of information from another modality into the reconstruction process. Our method is inspired by a novel iterative algorithm for coupled convolutional sparse coding, resulting in an interpretable network by design. We apply our model to the super-resolution of near-infrared image guided by RGB images. Experimental results show that our model outperforms state-of-the-art methods.

[1]  Yann LeCun,et al.  Learning Fast Approximations of Sparse Coding , 2010, ICML.

[2]  Pier Luigi Dragotti,et al.  Deep Coupled ISTA Network for Multi-Modal Image Super-Resolution , 2020, IEEE Transactions on Image Processing.

[3]  Pier Luigi Dragotti,et al.  Deep Convolutional Neural Network for Multi-Modal Image Restoration and Fusion , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Raja Giryes,et al.  Learned Convolutional Sparse Coding , 2017, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[5]  Thomas S. Huang,et al.  Robust Single Image Super-Resolution via Deep Networks With Sparse Prior , 2016, IEEE Transactions on Image Processing.

[6]  Miguel R. D. Rodrigues,et al.  Compressed Sensing With Prior Information: Strategies, Geometry, and Bounds , 2017, IEEE Transactions on Information Theory.

[7]  Iman Marivani,et al.  Multimodal Image Super-resolution via Deep Unfolding with Side Information , 2019, 2019 27th European Signal Processing Conference (EUSIPCO).

[8]  Wei Wu,et al.  Feedback Network for Image Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Miguel R. D. Rodrigues,et al.  Multimodal Image Super-Resolution via Joint Sparse Representations Induced by Coupled Dictionaries , 2017, IEEE Transactions on Computational Imaging.

[10]  Narendra Ahuja,et al.  Deep Joint Image Filtering , 2016, ECCV.

[11]  Kaiqi Huang,et al.  Fast End-to-End Trainable Guided Filter , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[12]  Yueting Zhuang,et al.  Supervised Coupled Dictionary Learning with Group Structures for Multi-modal Retrieval , 2013, AAAI.

[13]  Iman Marivani,et al.  Learned Multimodal Convolutional Sparse Coding for Guided Image Super-Resolution , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[14]  Juhan Nam,et al.  Multimodal Deep Learning , 2011, ICML.

[15]  Andreas Krause,et al.  Advances in Neural Information Processing Systems (NIPS) , 2014 .

[16]  Evaggelia Tsiligianni,et al.  Deep Coupled-Representation Learning for Sparse Linear Inverse Problems With Side Information , 2019, IEEE Signal Processing Letters.

[17]  Bernard Ghanem,et al.  ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Iman Marivani,et al.  Multimodal Deep Unfolding for Guided Image Super-Resolution , 2020, IEEE Transactions on Image Processing.

[19]  Miguel R. D. Rodrigues,et al.  X-ray image separation via coupled dictionary learning , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[20]  Jonathan Le Roux,et al.  Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures , 2014, ArXiv.

[21]  Quan Pan,et al.  Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Jian Sun,et al.  Deep ADMM-Net for Compressive Sensing MRI , 2016, NIPS.