DeepDemosaicking: Adaptive Image Demosaicking via Multiple Deep Fully Convolutional Networks

Convolutional neural networks are currently the state-of-the-art solution for a wide range of image processing tasks. Their deep architecture extracts low- and high-level features from images, thus improving the model’s performance. In this paper, we propose a method for image demosaicking based on deep convolutional neural networks. Demosaicking is the task of reproducing full color images from incomplete images formed from overlaid color filter arrays on image sensors found in digital cameras. Instead of producing the output image directly, the proposed method divides the demosaicking task into an initial demosaicking step and a refinement step. The initial step produces a rough demosaicked image containing unwanted color artifacts. The refinement step then reduces these color artifacts using deep residual estimation and multi-model fusion producing a higher quality image. Experimental results show that the proposed method outperforms several existing and state-of-the-art methods in terms of both the subjective and objective evaluations.

[1]  Kai-Lung Hua,et al.  Context-aware joint dictionary learning for color image demosaicking , 2016, J. Vis. Commun. Image Represent..

[2]  Martin Vetterli,et al.  Demosaicking by Alternating Projections: Theory and Fast One-Step Implementation , 2010, IEEE Transactions on Image Processing.

[3]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[4]  Masatoshi Okutomi,et al.  Minimized-Laplacian residual interpolation for color image demosaicking , 2014, Electronic Imaging.

[5]  Yan Wang,et al.  DeepContour: A deep convolutional feature learned by positive-sharing loss for contour detection , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Mei-Chen Yeh,et al.  Artist-based Classification via Deep Learning with Multi-scale Weighted Pooling , 2016, ACM Multimedia.

[7]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[8]  Wilfried Philips,et al.  The Effect of the Color Filter Array Layout Choice on State-of-the-Art Demosaicing , 2019, Sensors.

[9]  Eric Dubois,et al.  Filter Design for Adaptive Frequency-Domain Bayer Demosaicking , 2006, 2006 International Conference on Image Processing.

[10]  Radu Timofte,et al.  Demosaicing Based on Directional Difference Regression and Efficient Regression Priors , 2016, IEEE Transactions on Image Processing.

[11]  Kai-Lung Hua,et al.  Self-learning approach to color demosaicking via support vector regression , 2012, 2012 19th IEEE International Conference on Image Processing.

[12]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[13]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Lei Zhang,et al.  Color demosaicking via directional linear minimum mean square-error estimation , 2005, IEEE Transactions on Image Processing.

[15]  Yuk-Hee Chan,et al.  Enhanced integrated gradient and its applications to color demosaicing , 2012, 2012 IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC 2012).

[16]  Rastislav Lukac,et al.  2004 International Conference on Image Processing (ICIP) A NORMALIZED MODEL FOR COLOR-RATIO BASED DEMOSAICKING SCHEMES , 2022 .

[17]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[18]  Yücel Altunbasak,et al.  Gradient based threshold free color filter array interpolation , 2010, 2010 IEEE International Conference on Image Processing.

[19]  Lei Zhang,et al.  Color demosaicking by local directional interpolation and nonlocal adaptive thresholding , 2011, J. Electronic Imaging.

[20]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[21]  Wangmeng Zuo,et al.  COLOR IMAGE DEMOSAICKING VIA DEEP RESIDUAL LEARNING , 2017 .

[22]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[23]  Michael Elad,et al.  Sparse Representation for Color Image Restoration , 2008, IEEE Transactions on Image Processing.

[24]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.

[25]  Ron Kimmel,et al.  Demosaicing: Image Reconstruction from Color CCD Samples , 1998, ECCV.

[26]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[27]  Yi-Nung Liu,et al.  Color Filter Array Demosaicking Using Self-validation Framework , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[28]  Soo-Chang Pei,et al.  Effective color interpolation in CCD color filter arrays using signal correlation , 2003, IEEE Trans. Circuits Syst. Video Technol..

[29]  Chung-Yen Su Highly effective iterative demosaicing using weighted-edge and color-difference interpolations , 2006, IEEE Transactions on Consumer Electronics.

[30]  Henrique S. Malvar,et al.  High-quality linear interpolation for demosaicing of Bayer-patterned color images , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[31]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[32]  Masatoshi Okutomi,et al.  Adaptive residual interpolation for color image demosaicking , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[33]  Hiroshi Ishikawa,et al.  Let there be color! , 2016, ACM Trans. Graph..

[34]  Yi-Qing Wang,et al.  A multilayer neural network for image demosaicking , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[35]  Sabine Süsstrunk,et al.  Linear demosaicing inspired by the human visual system , 2005, IEEE Transactions on Image Processing.

[36]  Wen-Huang Cheng,et al.  Computer-aided classification of lung nodules on computed tomography images via deep learning technique , 2015, OncoTargets and therapy.

[37]  한보형,et al.  Learning Deconvolution Network for Semantic Segmentation , 2015 .

[38]  Tamara Seybold,et al.  Beyond the Kodak image set: A new reference set of color image sequences , 2013, 2013 IEEE International Conference on Image Processing.

[39]  Thomas W. Parks,et al.  Adaptive homogeneity-directed demosaicing algorithm , 2005, IEEE Transactions on Image Processing.

[40]  Masatoshi Okutomi,et al.  Residual interpolation for color image demosaicking , 2013, 2013 IEEE International Conference on Image Processing.

[41]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Kathiravan Srinivasan,et al.  Robust RGB-D Hand Tracking Using Deep Learning Priors , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[43]  Luc Van Gool,et al.  Jointly Optimized Regressors for Image Super‐resolution , 2015, Comput. Graph. Forum.

[44]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Masatoshi Okutomi,et al.  Beyond Color Difference: Residual Interpolation for Color Image Demosaicking , 2016, IEEE Transactions on Image Processing.

[46]  Yi-Nung Liu,et al.  A no -reference quality evaluation method for CFA Demosaicking , 2010, 2010 Digest of Technical Papers International Conference on Consumer Electronics (ICCE).

[47]  Frédo Durand,et al.  Deep joint demosaicking and denoising , 2016, ACM Trans. Graph..

[48]  Wei Ye,et al.  Color Image Demosaicing Using Iterative Residual Interpolation , 2015, IEEE Transactions on Image Processing.

[49]  Guillermo Sapiro,et al.  Non-local sparse models for image restoration , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[50]  Eric Dubois,et al.  Frequency-domain methods for demosaicking of Bayer-sampled color images , 2005, IEEE Signal Processing Letters.

[51]  Antoni Buades,et al.  A Demosaicking Algorithm with Adaptive Inter-Channel Correlation , 2015, Image Process. Line.

[52]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).