SCGAN: Saliency Map-Guided Colorization With Generative Adversarial Network

Given a grayscale photograph, the colorization system estimates a visually plausible colorful image. Conventional methods often use semantics to colorize grayscale images. However, in these methods, only classification semantic information is embedded, resulting in semantic confusion and color bleeding in the final colorized image. To address these issues, we propose a fully automatic Saliency Map-guided Colorization with Generative Adversarial Network (SCGAN) framework. It jointly predicts the colorization and saliency map to minimize semantic confusion and color bleeding in the colorized image. Since the global features from pre-trained VGG-16-Gray network are embedded to the colorization encoder, the proposed SCGAN can be trained with much less data than state-of-the-art methods to achieve perceptually reasonable colorization. In addition, we propose a novel saliency map-based guidance method. Branches of the colorization decoder are used to predict the saliency map as a proxy target. Moreover, two hierarchical discriminators are utilized for the generated colorization and saliency map, respectively, in order to strengthen visual perception performance. The proposed system is evaluated on ImageNet validation set. Experimental results show that SCGAN can generate more reasonable colorized images than state-of-the-art techniques.

[1]  Tam V. Nguyen,et al.  Salient Object Detection via Augmented Hypotheses , 2015, IJCAI.

[2]  Aditya Deshpande,et al.  Learning Diverse Image Colorization , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Gregory Shakhnarovich,et al.  Colorization as a Proxy Task for Visual Understanding , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Deepu Rajan,et al.  Image colorization using similar images , 2012, ACM Multimedia.

[5]  Qingming Huang,et al.  Image Saliency Detection Video Saliency Detection Co-saliency Detection Temporal RGBD Saliency Detection Motion , 2018 .

[6]  Klaus Mueller,et al.  Transferring color to greyscale images , 2002, ACM Trans. Graph..

[7]  Xiaochun Cao,et al.  Fake Colorized Image Detection , 2018, IEEE Transactions on Information Forensics and Security.

[8]  Liqing Zhang,et al.  Saliency Detection: A Spectral Residual Approach , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Lorenzo Torresani,et al.  Multiple hypothesis colorization and its application to image compression , 2017, Comput. Vis. Image Underst..

[10]  Huchuan Lu,et al.  Saliency Detection via Graph-Based Manifold Ranking , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Stephen Lin,et al.  Semantic colorization with internet images , 2011, ACM Trans. Graph..

[12]  Stephen Lin,et al.  Intrinsic colorization , 2008, ACM Trans. Graph..

[13]  Larry S. Davis,et al.  Submodular Salient Region Detection , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Tam V. Nguyen,et al.  Semantic Prior Analysis for Salient Object Detection , 2019, IEEE Transactions on Image Processing.

[15]  Namil Kim,et al.  Multispectral pedestrian detection: Benchmark dataset and baseline , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Naila Murray,et al.  Saliency estimation using a non-parametric low-level vision model , 2011, CVPR 2011.

[17]  Junwei Han,et al.  DHSNet: Deep Hierarchical Saliency Network for Salient Object Detection , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Li Xu,et al.  A sparse control model for image and video editing , 2013, ACM Trans. Graph..

[19]  Edgar Simo-Serra,et al.  DeepRemaster , 2019, ACM Trans. Graph..

[20]  Tieyong Zeng,et al.  A Superpixel-Based Variational Model for Image Colorization , 2020, IEEE Transactions on Visualization and Computer Graphics.

[21]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[22]  Zhenyang Wu,et al.  Natural color image enhancement and evaluation algorithm based on human visual system , 2006, Comput. Vis. Image Underst..

[23]  Gregory Shakhnarovich,et al.  Learning Representations for Automatic Colorization , 2016, ECCV.

[24]  Ting Zhao,et al.  Pyramid Feature Attention Network for Saliency Detection , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Harry Shum,et al.  Natural Image Colorization , 2007, Rendering Techniques.

[26]  Eunhyeok Park,et al.  Tag2Pix: Line Art Colorization Using Text Tag With SECat and Changing Loss , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[27]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[28]  Xiang Zhang,et al.  Superpixel-Based Spatiotemporal Saliency Detection , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[29]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[30]  Hiroshi Ishikawa,et al.  Let there be color! , 2016, ACM Trans. Graph..

[31]  Dani Lischinski,et al.  Colorization using optimization , 2004, ACM Trans. Graph..

[32]  Ling Shao,et al.  Pixel-level Semantics Guided Image Colorization , 2018, BMVC.

[33]  Yong Yu,et al.  Unsupervised Diverse Colorization via Generative Adversarial Networks , 2017, ECML/PKDD.

[34]  David A. Forsyth,et al.  Learning Large-Scale Automatic Image Colorization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[35]  David A. Forsyth,et al.  Structural Consistency and Controllability for Diverse Colorization , 2018, ECCV.

[36]  Shi-Min Hu,et al.  Global contrast based salient region detection , 2011, CVPR 2011.

[37]  Andrew L. Maas Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[38]  Abdelrahman Eldesokey,et al.  Unpaired Thermal to Visible Spectrum Transfer Using Adversarial Training , 2018, ECCV Workshops.

[39]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[40]  Amine Bermak,et al.  Deep Exemplar-Based Video Colorization , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Aykut Erdem,et al.  Visual saliency estimation by nonlinearly integrating features using region covariances. , 2013, Journal of vision.

[42]  Chi-Keung Tang,et al.  Local color transfer via probabilistic segmentation by expectation-maximization , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[43]  Junting Pan,et al.  SalGAN: visual saliency prediction with adversarial networks , 2017 .

[44]  Qi Zhao,et al.  Attentive Systems: A Survey , 2017, International Journal of Computer Vision.

[45]  Coloma Ballester,et al.  ChromaGAN: Adversarial Picture Colorization with Semantic Class Distribution , 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[46]  Tiantian Wang,et al.  A Multistage Refinement Network for Salient Object Detection , 2020, IEEE Transactions on Image Processing.

[47]  Sabine Süsstrunk,et al.  Measuring colorfulness in natural images , 2003, IS&T/SPIE Electronic Imaging.

[48]  Xiaowu Chen,et al.  Manifold preserving edit propagation , 2012, ACM Trans. Graph..

[49]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[50]  Steven C. H. Hoi,et al.  Salient Object Detection With Pyramid Attention and Salient Edges , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Huchuan Lu,et al.  Saliency Detection with Recurrent Fully Convolutional Networks , 2016, ECCV.

[52]  Mohammad Norouzi,et al.  PixColor: Pixel Recursive Colorization , 2017, BMVC.

[53]  Bo Li,et al.  Example-Based Image Colorization Using Locality Consistent Sparse Representation , 2017, IEEE Transactions on Image Processing.

[54]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[55]  Huchuan Lu,et al.  Deep networks for saliency detection via local estimation and global search , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  John K. Tsotsos,et al.  Saliency Based on Information Maximization , 2005, NIPS.

[57]  Tien-Tsin Wong,et al.  Two-stage sketch colorization , 2018, ACM Trans. Graph..

[58]  Hanqiu Sun,et al.  Video Colorization Using Parallel Optimization in Feature Space , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[59]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[60]  Huchuan Lu,et al.  Multi-Scale Interactive Network for Salient Object Detection , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[61]  Jianmin Jiang,et al.  A Simple Pooling-Based Design for Real-Time Salient Object Detection , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Ruigang Yang,et al.  Saliency-Aware Video Object Segmentation , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[63]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[64]  Saumik Bhattacharya,et al.  Spatiotemporal Colorization of Video Using 3D Steerable Pyramids , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[65]  Bin Sheng,et al.  Deep Colorization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[66]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[67]  Jun-Cheng Chen,et al.  An adaptive edge detection based colorization algorithm and its applications , 2005, ACM Multimedia.

[68]  Aurélie Bugeau,et al.  Variational Exemplar-Based Image Colorization , 2014, IEEE Transactions on Image Processing.

[69]  Zhuowen Tu,et al.  Deeply Supervised Salient Object Detection with Short Connections , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[70]  Xiaojuan Ma,et al.  Coloring with Words: Guiding Image Colorization Through Text-Based Palette Generation , 2018, ECCV.

[71]  Christoph H. Lampert,et al.  Probabilistic Image Colorization , 2017, BMVC.

[72]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[73]  Bolei Zhou,et al.  Places: A 10 Million Image Database for Scene Recognition , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[74]  Bernhard Schölkopf,et al.  Automatic Image Colorization Via Multimodal Predictions , 2008, ECCV.

[75]  Lihi Zelnik-Manor,et al.  The Contextual Loss for Image Transformation with Non-Aligned Data , 2018, ECCV.

[76]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[77]  Michael Felsberg,et al.  Generating Visible Spectrum Images from Thermal Infrared , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[78]  Alexei A. Efros,et al.  Real-time user-guided image colorization with learned deep priors , 2017, ACM Trans. Graph..

[79]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[80]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[81]  Stan Sclaroff,et al.  Saliency Detection: A Boolean Map Approach , 2013, 2013 IEEE International Conference on Computer Vision.

[82]  Sebastian Nowozin,et al.  f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization , 2016, NIPS.

[83]  Bing Yu,et al.  Automatic Colorization of Images from Chinese Black and White Films Based on CNN , 2018, 2018 International Conference on Audio, Language and Image Processing (ICALIP).

[84]  Ling Shao,et al.  Pixelated Semantic Colorization , 2019, International Journal of Computer Vision.

[85]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[86]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[87]  Pietro Perona,et al.  Graph-Based Visual Saliency , 2006, NIPS.

[88]  Guillermo Sapiro,et al.  Fast image and video colorization using chrominance blending , 2006, IEEE Transactions on Image Processing.

[89]  Shi-Min Hu,et al.  Efficient affinity-based edit propagation using K-D tree , 2009, ACM Trans. Graph..

[90]  Qifeng Chen,et al.  Fully Automatic Video Colorization With Self-Regularization and Diversity , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[91]  Yizhou Yu,et al.  Visual saliency based on multiscale deep features , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[92]  Chengying Gao,et al.  Language-based colorization of scene sketches , 2019, ACM Trans. Graph..

[93]  Gang Wang,et al.  Progressive Attention Guided Recurrent Network for Salient Object Detection , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[94]  Chunping Hou,et al.  Blind Quality Assessment of Tone-Mapped Images Considering Colorfulness, Naturalness, and Structure , 2019, IEEE Transactions on Industrial Electronics.

[95]  Dani Lischinski,et al.  Colorization by example , 2005, EGSR '05.

[96]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[97]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[98]  Tien-Tsin Wong,et al.  Manga colorization , 2006, ACM Trans. Graph..

[99]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.