Double-Channel Guided Generative Adversarial Network for Image Colorization

Image colorization has a widespread application in video and image restoration in the past few years. Recently, automatic colorization methods based on deep learning have shown impressive performance. However, these methods map grayscale image input into multi-channel output directly. In the process, it usually loses detailed information during feature extraction, resulting in abnormal colors in local areas of the colorization image. To overcome abnormal colors and improve colorization quality, we propose a novel Double-Channel Guided Generative Adversarial Network (DCGGAN). It includes two modules: a reference component matching module and a double-channel guided colorization module. The reference component matching module is introduced to select suitable reference color components as auxiliary information of the input. The double-channel guided colorization module is designed to learn the mapping relationship from the grayscale to each color channel with the assistance of reference color components. Experimental results show that the proposed DCGGAN outperforms existing methods on different quality metrics and achieves state-of-the-art performance.

[1]  Dani Lischinski,et al.  Colorization by example , 2005, EGSR '05.

[2]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Qiliang Du,et al.  An Enhanced Image Fusion Algorithm by Combined Histogram Equalization and Fast Gray Level Grouping Using Multi-Scale Decomposition and Gray-PCA , 2020, IEEE Access.

[4]  Guillermo Sapiro,et al.  Fast image and video colorization using chrominance blending , 2006, IEEE Transactions on Image Processing.

[5]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[6]  Michael K. Ng,et al.  Image colorization by fusion of color transfers based on DFT and variance features , 2019, Comput. Math. Appl..

[7]  Harry Shum,et al.  Natural Image Colorization , 2007, Rendering Techniques.

[8]  Giovanni Emanuele Corazza,et al.  PSNR evaluation and alignment recovery for mobile satellite video broadcasting , 2012, 2012 6th Advanced Satellite Multimedia Systems Conference (ASMS) and 12th Signal Processing for Space Communications Workshop (SPSC).

[9]  David A. Forsyth,et al.  Structural Consistency and Controllability for Diverse Colorization , 2018, ECCV.

[10]  Qifeng Chen,et al.  Fully Automatic Video Colorization With Self-Regularization and Diversity , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[12]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[13]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[14]  Stephen Lin,et al.  Intrinsic colorization , 2008, ACM Trans. Graph..

[15]  Dani Lischinski,et al.  Colorization using optimization , 2004, ACM Trans. Graph..

[16]  Yong Seok Heo,et al.  Probabilistic Gaussian similarity-based local colour transfer , 2016 .

[17]  Yuhao Wang,et al.  Joint Intensity-Gradient Guided Generative Modeling for Colorization , 2020, The Visual Computer.

[18]  Alexei A. Efros,et al.  Real-time user-guided image colorization with learned deep priors , 2017, ACM Trans. Graph..

[19]  Ling Shao,et al.  Pixelated Semantic Colorization , 2019, International Journal of Computer Vision.

[20]  Gregory Shakhnarovich,et al.  Learning Representations for Automatic Colorization , 2016, ECCV.

[21]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.

[22]  Anxiang Zeng,et al.  Vectorization Based Color Transfer for Portrait Images , 2019, Comput. Aided Des..

[23]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[24]  Tien-Tsin Wong,et al.  Manga colorization , 2006, ACM Trans. Graph..

[25]  Nima Tajbakhsh,et al.  UNet++: A Nested U-Net Architecture for Medical Image Segmentation , 2018, DLMIA/ML-CDS@MICCAI.

[26]  Bhawani Shankar,et al.  Multi Focus Image Fusion using Combined Median and Average Filter based Hybrid Stationary Wavelet Transform and Principal Component Analysis , 2018 .

[27]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[28]  Lai-Man Po,et al.  SCGAN: Saliency Map-Guided Colorization With Generative Adversarial Network , 2020, IEEE Transactions on Circuits and Systems for Video Technology.

[29]  Ling Shao,et al.  Pixel-level Semantics Guided Image Colorization , 2018, BMVC.

[30]  Alberto Del Bimbo,et al.  Image and Video Restoration and Compression Artefact Removal Using a NoGAN Approach , 2020, ACM Multimedia.

[31]  Hung-Kuo Chu,et al.  Instance-Aware Image Colorization , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Deepu Rajan,et al.  Image colorization using similar images , 2012, ACM Multimedia.

[33]  Stephen Lin,et al.  Semantic colorization with internet images , 2011, ACM Trans. Graph..

[34]  Ira Kemelmacher-Shlizerman,et al.  The MegaFace Benchmark: 1 Million Faces for Recognition at Scale , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[36]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[37]  Jun-Cheng Chen,et al.  An adaptive edge detection based colorization algorithm and its applications , 2005, ACM Multimedia.

[38]  S. Amirhassan Monadjemi,et al.  Film Colorization, Using Artificial Neural Networks and Laws Filters , 2010, J. Comput..

[39]  Tobias Weyand,et al.  Google Landmarks Dataset v2 – A Large-Scale Benchmark for Instance-Level Recognition and Retrieval , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Klaus Mueller,et al.  Transferring color to greyscale images , 2002, ACM Trans. Graph..