Interactive Deep Colorization Using Simultaneous Global and Local Inputs

Colorization methods using deep neural networks have become a recent trend. However, most of them do not allow user inputs, or only allow limited user inputs (only global inputs or only local inputs), to control the output colorful images. The possible reason is that it’s difficult to differentiate the influence of different kind of user inputs in network training. To solve this problem, we propose a novel deep colorization method allowing inputting global and local inputs simultaneously or individually, which is not supported in previous deep colorization methods. The key steps include designing a neural network model that can appropriately combine the different inputs, and designing an appropriate loss function that can differentiate the influence of different inputs. Experimental results show that our method can magnificently control the colorized results and generate state-of-art results.

[1]  Masayuki Nakajima,et al.  Example-Based Color Stylization of Images , 2005, TAP.

[2]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[3]  Michal Kawulok,et al.  Textural Features for Scribble-Based Image Colorization , 2011, Computer Recognition Systems 4.

[4]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[5]  Hendrik P. A. Lensch,et al.  Infrared Colorization Using Deep Convolutional Neural Networks , 2016, 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA).

[6]  Yan Zhang,et al.  Image Colorization Using Convolutional Neural Network , 2016, IGTA.

[7]  Bernhard Schölkopf,et al.  Automatic Image Colorization Via Multimodal Predictions , 2008, ECCV.

[8]  Stephen Lin,et al.  Semantic colorization with internet images , 2011, ACM Trans. Graph..

[9]  Kevin Frans,et al.  Outline Colorization through Tandem Adversarial Networks , 2017, ArXiv.

[10]  Jun-Cheng Chen,et al.  An adaptive edge detection based colorization algorithm and its applications , 2005, ACM Multimedia.

[11]  Xiaowu Chen,et al.  Manifold preserving edit propagation , 2012, ACM Trans. Graph..

[12]  Li Chen,et al.  Image colorization using Bayesian nonlocal inference , 2011, J. Electronic Imaging.

[13]  Shi-Min Hu,et al.  Efficient affinity-based edit propagation using K-D tree , 2009, ACM Trans. Graph..

[14]  Chun Chen,et al.  Data-driven image color theme enhancement , 2010, SIGGRAPH 2010.

[15]  Bin Sheng,et al.  Deep Colorization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[16]  Harry Shum,et al.  Natural Image Colorization , 2007, Rendering Techniques.

[17]  Dongdong Chen,et al.  Deep exemplar-based colorization , 2018, ACM Trans. Graph..

[18]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Guillermo Sapiro,et al.  Fast image and video colorization using chrominance blending , 2006, IEEE Transactions on Image Processing.

[20]  Edward H. Adelson,et al.  Eurographics Symposium on Rendering 2008 Scribbleboost: Adding Classification to Edge-aware Interpolation of Local Image and Video Adjustments , 2022 .

[21]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Alexei A. Efros,et al.  Real-time user-guided image colorization with learned deep priors , 2017, ACM Trans. Graph..

[23]  Hiroshi Ishikawa,et al.  Let there be color! , 2016, ACM Trans. Graph..

[24]  Klaus Mueller,et al.  Transferring color to greyscale images , 2002, ACM Trans. Graph..

[25]  Szymon Rusinkiewicz,et al.  AutoStyle: Automatic Style Transfer from Image Collections to Users' Images , 2014, Comput. Graph. Forum.

[26]  Yoshihiro Kanamori,et al.  DeepProp: Extracting Deep Features from a Single Image for Edit Propagation , 2016, Comput. Graph. Forum.

[27]  Fisher Yu,et al.  Scribbler: Controlling Deep Image Synthesis with Sketch and Color , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Gregory Shakhnarovich,et al.  Learning Representations for Automatic Colorization , 2016, ECCV.

[29]  S. V. N. Vishwanathan,et al.  Learning to compress images and videos , 2007, ICML '07.

[30]  Deepu Rajan,et al.  Image colorization using similar images , 2012, ACM Multimedia.

[31]  Yizhou Yu,et al.  Automatic Photo Adjustment Using Deep Neural Networks , 2014, ACM Trans. Graph..

[32]  Michal Kawulok,et al.  Image colorization with competitive propagation paths and chrominance blending , 2010 .

[33]  Dani Lischinski,et al.  Colorization using optimization , 2004, SIGGRAPH 2004.

[34]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[35]  Stephen DiVerdi,et al.  Palette-based photo recoloring , 2015, ACM Trans. Graph..

[36]  Takahiko Horiuchi,et al.  Colorization for Monochrome Image with Texture , 2005, Color Imaging Conference.

[37]  David A. Forsyth,et al.  Learning Large-Scale Automatic Image Colorization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[38]  Ruck Thawonmas,et al.  Image Colorization Using a Deep Convolutional Neural Network , 2016, ArXiv.

[39]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[40]  Stephen Lin,et al.  Intrinsic colorization , 2008, SIGGRAPH 2008.

[41]  K. Sasaki,et al.  Learning to simplify , 2016, ACM Trans. Graph..

[42]  Domonkos Varga,et al.  Automatic Cartoon Colorization Based on Convolutional Neural Network , 2017, CBMI.

[43]  Domonkos Varga,et al.  Fully automatic image colorization based on Convolutional Neural Network , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).