ConvUNeXt: A Lightweight Convolutional Neural Network for Watercolor Image Translation

Image-to-image transformation is the task of transforming an image from one domain to another. It includes the task of converting an image to an artistic style such as oil painting, tile art, and watercolor. These conversion techniques have rapidly advanced with the development of machine learning in recent years. This paper proposes a method for generating watercolor paintings using machine learning technologies. The proposed method uses conditional generative adversary networks (cGANs) to convert the style of input images from real to watercolor. Specifically, we propose a ConvUNeXt generator that generates high-quality watercolor images while preserving the color information of the original image. It employs an encoder-decoder structure with skip connections and introduces multiple ConvNeXt blocks to learn image features. Experimental results show that the proposed model significantly outperforms traditional rendering methods in the time consumption of image translation while achieving comparable translation performance.

[1]  Junyan Zhu,et al.  On Aliased Resizing and Surprising Subtleties in GAN Evaluation , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Trevor Darrell,et al.  A ConvNet for the 2020s , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Qi Tian,et al.  Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation , 2021, ECCV Workshops.

[4]  K. Nakano,et al.  A GPU Implementation of Watercolor Painting Image Generation , 2021, 2021 Ninth International Symposium on Computing and Networking Workshops (CANDARW).

[5]  Lei Zhang,et al.  High-Resolution Photorealistic Image Translation in Real-Time: A Laplacian Pyramid Translation Network , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Koji Nakano,et al.  Tile Art Image Generation Using Conditional Generative Adversarial Networks , 2018, 2018 Sixth International Symposium on Computing and Networking Workshops (CANDARW).

[7]  Ling-Yu Duan,et al.  ChipGAN: A Generative Adversarial Network for Chinese Ink Wash Painting Style Transfer , 2018, ACM Multimedia.

[8]  Ming Wu,et al.  D-LinkNet: LinkNet with Pretrained Encoder and Dilated Convolution for High Resolution Satellite Imagery Road Extraction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[9]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[10]  Jan Kautz,et al.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[11]  Eugenio Culurciello,et al.  LinkNet: Exploiting encoder representations for efficient semantic segmentation , 2017, 2017 IEEE Visual Communications and Image Processing (VCIP).

[12]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[13]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Vincent Dumoulin,et al.  Deconvolution and Checkerboard Artifacts , 2016 .

[17]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[18]  Kevin Gimpel,et al.  Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units , 2016, ArXiv.

[19]  Thomas Brox,et al.  Generating Images with Perceptual Similarity Metrics based on Deep Networks , 2016, NIPS.

[20]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[22]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[23]  Andrew L. Maas Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[24]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[25]  David Salesin,et al.  Computer-generated watercolor , 1997, SIGGRAPH.