Line Art Colorization with Concatenated Spatial Attention

Line art plays a fundamental role in illustration and design, and allows for iteratively polishing designs. However, as they lack color, they can have issues in conveying final designs. In this work, we propose an interactive colorization approach based on a conditional generative adversarial network that takes both the line art and color hints as inputs to produce a high-quality colorized image. Our approach is based on a U-net architecture with a multi-discriminator framework. We propose a Concatenation and Spatial Attention module that is able to generate more consistent and higher quality of line art colorization from user given hints. We evaluate on a large-scale illustration dataset and comparison with existing approaches corroborate the effectiveness of our approach.

[1]  Jaegul Choo,et al.  Reference-Based Sketch Image Colorization Using Augmented-Self Reference and Dense Semantic Correspondence , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  John Dingliana,et al.  LazyBrush: Flexible Painting Tool for Hand‐drawn Cartoons , 2009, Comput. Graph. Forum.

[3]  Mario Lucic,et al.  Are GANs Created Equal? A Large-Scale Study , 2017, NeurIPS.

[4]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[5]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[6]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[7]  Eunhyeok Park,et al.  Tag2Pix: Line Art Colorization Using Text Tag With SECat and Changing Loss , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[8]  Zengchang Qin,et al.  Auto-painter: Cartoon image generation from sketch by using conditional Wasserstein generative adversarial networks , 2018, Neurocomputing.

[9]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Haojie Li,et al.  User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks , 2018, ACM Multimedia.

[11]  Holger Winnemöller,et al.  XDoG: An eXtended difference-of-Gaussians compendium including advanced image stylization , 2012, Comput. Graph..

[12]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[13]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[14]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Alexei A. Efros,et al.  Real-time user-guided image colorization with learned deep priors , 2017, ACM Trans. Graph..

[16]  Kazuyuki Hiroshiba,et al.  Comicolorization: semi-automatic manga colorization , 2017, SIGGRAPH Asia Technical Briefs.

[17]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Jan Kautz,et al.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[19]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[20]  Tien-Tsin Wong,et al.  Manga colorization , 2006, ACM Trans. Graph..

[21]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[22]  Hiroshi Ishikawa,et al.  Mastering Sketching: Adversarial Augmentation for Structured Prediction , 2017 .

[23]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Kevin Frans,et al.  Outline Colorization through Tandem Adversarial Networks , 2017, ArXiv.

[25]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.