HistoGAN: Controlling Colors of GAN-Generated and Real Images via Color Histograms

While generative adversarial networks (GANs) can successfully produce high-quality images, they can be challenging to control. Simplifying GAN-based image generation is critical for their adoption in graphic design and artistic work. This goal has led to significant interest in methods that can intuitively control the appearance of images generated by GANs. In this paper, we present HistoGAN, a color histogram-based method for controlling GAN-generated images’ colors. We focus on color histograms as they provide an intuitive way to describe image color while remaining decoupled from domain-specific semantics. Specifically, we introduce an effective modification of the recent StyleGAN architecture [31] to control the colors of GAN-generated images specified by a target color histogram feature. We then describe how to expand HistoGAN to recolor real images. For image recoloring, we jointly train an encoder network along with HistoGAN. The recoloring model, ReHistoGAN, is an unsupervised approach trained to encourage the network to keep the original image’s content while changing the colors based on the given target histogram. We show that this histogram-based approach offers a better way to control GAN-generated and real images’ colors while producing more compelling results compared to existing alternative strategies.

[1]  Sylvain Paris,et al.  Learning photographic global tonal adjustment with a database of input/output image pairs , 2011, CVPR 2011.

[2]  Andrew Zisserman,et al.  Automated Flower Classification over a Large Number of Classes , 2008, 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing.

[3]  Aditya Deshpande,et al.  Learning Diverse Image Colorization , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  J. Uijlings,et al.  The Open Images Dataset V4 , 2018, International Journal of Computer Vision.

[5]  Michael S. Brown,et al.  Image Recoloring Based on Object Color Distributions , 2019, Eurographics.

[6]  Tero Karras,et al.  Analyzing and Improving the Image Quality of StyleGAN , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Dimitris Samaras,et al.  Wasserstein GAN With Quadratic Transport Cost , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[8]  Stephen DiVerdi,et al.  Palette-based photo recoloring , 2015, ACM Trans. Graph..

[9]  Michael S. Brown,et al.  Learning to Correct Overexposed and Underexposed Photos , 2020, ArXiv.

[10]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[11]  Bolei Zhou,et al.  Scene Parsing through ADE20K Dataset , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Chunxia Xiao,et al.  Palette-Based Image Recoloring Using Color Decomposition Optimization , 2017, IEEE Transactions on Image Processing.

[13]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Andrew L. Maas.  Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[15]  Kate Saenko,et al.  COCO-FUNIT: Few-Shot Unsupervised Image Translation with a Content Conditioned Style Encoder , 2020, ECCV.

[16]  Bidyut Baran Chaudhuri,et al.  diffGrad: An Optimization Method for Convolutional Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[17]  G D Finlayson,et al.  Color constancy at a pixel. , 2001, Journal of the Optical Society of America. A, Optics, image science, and vision.

[18]  Elli Angelopoulou,et al.  The importance of the normalizing channel in log-chromaticity space , 2012, 2012 19th IEEE International Conference on Image Processing.

[19]  Michael S. Brown,et al.  Illuminant Aware Gamut‐Based Color Transfer , 2014, Comput. Graph. Forum.

[20]  Yinda Zhang,et al.  LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop , 2015, ArXiv.

[21]  Michael S. Brown,et al.  When Color Constancy Goes Wrong: Correcting Improperly White-Balanced Images , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Yong Jae Lee,et al.  MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation , 2019, Computer Vision and Pattern Recognition.

[23]  Jung-Woo Ha,et al.  StarGAN v2: Diverse Image Synthesis for Multiple Domains , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[25]  Michael S. Brown,et al.  Learning Multi-Scale Photo Exposure Correction , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Peter Wonka,et al.  Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space? , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[27]  Anil Kokaram,et al.  The linear Monge-Kantorovitch linear colour mapping for example-based colour transfer , 2007 .

[28]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[29]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[30]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[31]  Stanislav Pidhorskyi,et al.  Adversarial Latent Autoencoders , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[33]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[34]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[35]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Lizhuang Ma,et al.  Color transfer in correlated color space , 2006, VRCIA '06.

[37]  Michael S. Brown,et al.  Sensor-Independent Illumination Estimation for DNN Models , 2019, BMVC.

[38]  Pierre Alliez,et al.  Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[39]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[40]  Bolei Zhou,et al.  Semantic Understanding of Scenes Through the ADE20K Dataset , 2016, International Journal of Computer Vision.

[41]  Frédo Durand,et al.  Data-driven hallucination of different times of day from a single outdoor photo , 2013, ACM Trans. Graph..

[42]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Sylvain Paris,et al.  Deep Photo Style Transfer , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Mahmoud Afifi,et al.  What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[45]  N. Mitra,et al.  StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows , 2020, ACM Trans. Graph..

[46]  Erik Reinhard,et al.  Colour Mapping: A Review of Recent Methods, Extensions and Applications , 2016, Comput. Graph. Forum.

[47]  Tammy Riklin-Raviv,et al.  DeepHist: Differentiable Joint and Color Histogram Layers for Image-to-Image Translation , 2020, ArXiv.

[48]  Dongdong Chen,et al.  Progressive Color Transfer With Dense Semantic Correspondences , 2017, ACM Trans. Graph..

[49]  Sanja Fidler,et al.  Nonlinear color triads for approximation, learning and direct manipulation of color distributions , 2020, ACM Trans. Graph..

[50]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[51]  Pietro Perona,et al.  Caltech-UCSD Birds 200 , 2010 .

[52]  Jonathan Krause,et al.  3D Object Representations for Fine-Grained Categorization , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[53]  Alexei A. Efros,et al.  Real-time user-guided image colorization with learned deep priors , 2017, ACM Trans. Graph..

[54]  Fei-Fei Li,et al.  Novel Dataset for Fine-Grained Image Categorization : Stanford Dogs , 2012 .

[55]  Jiawen Chen,et al.  Bilateral guided upsampling , 2016, ACM Trans. Graph..

[56]  Pietro Perona,et al.  The Caltech-UCSD Birds-200-2011 Dataset , 2011 .

[57]  Victor Lempitsky,et al.  High-Resolution Daytime Translation Without Domain Labels , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[58]  Tali Dekel,et al.  SinGAN: Learning a Generative Model From a Single Natural Image , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[59]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  Khaled F. Hussain,et al.  MPB: A modified Poisson blending technique , 2015, Computational Visual Media.

[61]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[62]  William T. Freeman,et al.  Semantic Pyramid for Image Generation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  Jonathan T. Barron,et al.  Convolutional Color Constancy , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[64]  Xiaogang Wang,et al.  Avatar-Net: Multi-scale Zero-Shot Style Transfer by Feature Decoration , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[65]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[66]  Leon A. Gatys,et al.  A Neural Algorithm of Artistic Style , 2015, ArXiv.

[67]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[68]  Mahmoud Afifi,et al.  11K Hands: Gender recognition and biometric identification using a large dataset of hand images , 2017, Multimedia Tools and Applications.

[69]  Xiaofeng Tao,et al.  Transient attributes for high-level understanding and editing of outdoor scenes , 2014, ACM Trans. Graph..