Photorealistic Style Transfer with Screened Poisson Equation

Recent work has shown impressive success in transferring painterly style to images. These approaches, however, fall short of photorealistic style transfer. Even when both the input and reference images are photographs, the output still exhibits distortions reminiscent of a painting. In this paper we propose an approach that takes as input a stylized image and makes it more photorealistic. It relies on the Screened Poisson Equation, maintaining the fidelity of the stylized image while constraining the gradients to those of the original input image. Our method is fast, simple, fully automatic and shows positive progress in making a stylized image photorealistic. Our results exhibit finer details and are less prone to artifacts than the state-of-the-art.

[1]  Michael Elad,et al.  Style Transfer Via Texture Synthesis , 2016, IEEE Transactions on Image Processing.

[2]  Xiaofeng Tao,et al.  Transient attributes for high-level understanding and editing of outdoor scenes , 2014, ACM Trans. Graph..

[3]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[4]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[5]  Matt J. Kusner,et al.  Deep Manifold Traversal: Changing Labels with Convolutional Features , 2015, ArXiv.

[6]  Xing Mei,et al.  Content‐Based Colour Transfer , 2013, Comput. Graph. Forum.

[7]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[8]  Dani Lischinski,et al.  A Closed-Form Solution to Natural Image Matting , 2008 .

[9]  Eli Shechtman,et al.  Image melding , 2012, ACM Trans. Graph..

[10]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[11]  Jonathon Shlens,et al.  A Learned Representation For Artistic Style , 2016, ICLR.

[12]  Chuan Li,et al.  Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Frédo Durand,et al.  Style transfer for headshot portraits , 2014, ACM Trans. Graph..

[14]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[15]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Lihi Zelnik-Manor,et al.  Saliency driven image manipulation , 2016, Machine Vision and Applications.

[17]  Serge J. Belongie,et al.  Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[18]  Sylvain Paris,et al.  Deep Photo Style Transfer , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Jean-Michel Morel,et al.  Screened Poisson Equation for Image Contrast Enhancement , 2014, Image Process. Line.

[20]  Baining Guo,et al.  Real-time texture synthesis by patch-based sampling , 2001, TOGS.

[21]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Zeev Farbman,et al.  Convolution pyramids , 2011, ACM Trans. Graph..

[23]  François Pitié,et al.  Automated colour grading using colour distribution transfer , 2007, Comput. Vis. Image Underst..

[24]  Michael F. Cohen,et al.  Fourier Analysis of the 2D Screened Poisson Equation for Gradient Domain Problems , 2008, ECCV.

[25]  Frédo Durand,et al.  Data-driven hallucination of different times of day from a single outdoor photo , 2013, ACM Trans. Graph..

[26]  Yair Weiss,et al.  Deriving intrinsic images from image sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[27]  Andrea Vedaldi,et al.  Texture Networks: Feed-forward Synthesis of Textures and Stylized Images , 2016, ICML.

[28]  Neus Sabater,et al.  Split and Match: Example-Based Adaptive Patch Sampling for Unsupervised Style Transfer , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[30]  Frédo Durand,et al.  Two-scale tone management for photographic look , 2006, SIGGRAPH 2006.

[31]  A.C. Kokaram,et al.  N-dimensional probability density function transfer and its application to color transfer , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[32]  Wojciech Matusik,et al.  Multi-scale image harmonization , 2010, SIGGRAPH 2010.

[33]  Mark W. Schmidt,et al.  Fast Patch-based Style Transfer of Arbitrary Style , 2016, ArXiv.