A survey of exemplar-based texture synthesis

Exemplar-based texture synthesis is the process of generating, from an input sample, new texture images of arbitrary size and which are perceptually equivalent to the sample. The two main approaches are statistics-based methods and patch re-arrangement methods. In the first class, a texture is characterized by a statistical signature; then, a random sampling conditioned to this signature produces genuinely different texture images. The second class boils down to a clever "copy-paste" procedure, which stitches together large regions of the sample. Hybrid methods try to combine ideas from both approaches to avoid their hurdles. The recent approaches using convolutional neural networks fit to this classification, some being statistical and others performing patch re-arrangement in the feature space. They produce impressive synthesis on various kinds of textures. Nevertheless, we found that most real textures are organized at multiple scales, with global structures revealed at coarse scales and highly varying details at finer ones. Thus, when confronted with large natural images of textures the results of state-of-the-art methods degrade rapidly, and the problem of modeling them remains wide open.

[1]  Michael Unser,et al.  Texture classification and segmentation using wavelet frames , 1995, IEEE Trans. Image Process..

[2]  Leon A. Gatys,et al.  What does it take to generate natural textures? , 2017, ICLR.

[3]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[4]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[5]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[6]  Song-Chun Zhu,et al.  Minimax Entropy Principle and Its Application to Texture Modeling , 1997, Neural Computation.

[7]  Olivier Alata,et al.  Non-stationary texture synthesis from random field modeling , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[8]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[9]  Bruno Galerne,et al.  Micro-Texture Synthesis by Phase Randomization , 2011, Image Process. Line.

[10]  Eero P. Simoncelli,et al.  A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients , 2000, International Journal of Computer Vision.

[11]  Andrea Vedaldi,et al.  Texture Networks: Feed-forward Synthesis of Textures and Stylized Images , 2016, ICML.

[12]  Roman Novak,et al.  Improving the Neural Algorithm of Artistic Style , 2016, ArXiv.

[13]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[14]  B. Galerne,et al.  The Heeger & Bergen Pyramid Based Texture Synthesis Algorithm , 2014, Image Process. Line.

[15]  Chuan Li,et al.  Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Jean-Michel Morel,et al.  Implementation of the "Non-Local Bayes" (NL-Bayes) Image Denoising Algorithm , 2013, Image Process. Line.

[17]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[18]  Jean-Michel Dischler,et al.  Multi-scale label-map extraction for texture synthesis , 2016, ACM Trans. Graph..

[19]  Jaakko Lehtinen,et al.  Reflectance modeling by neural texture synthesis , 2016, ACM Trans. Graph..

[20]  William T. Freeman,et al.  Presented at: 2nd Annual IEEE International Conference on Image , 1995 .

[21]  Alex Graves,et al.  Conditional Image Generation with PixelCNN Decoders , 2016, NIPS.

[22]  Béla Julesz,et al.  Visual Pattern Discrimination , 1962, IRE Trans. Inf. Theory.

[23]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Yann Gousseau,et al.  Variational Texture Synthesis with Sparsity and Spectrum Constraints , 2015, Journal of Mathematical Imaging and Vision.

[25]  Jean-Michel Morel,et al.  A Nonlocal Bayesian Image Denoising Algorithm , 2013, SIAM J. Imaging Sci..

[26]  B. Julesz Textons, the elements of texture perception, and their interactions , 1981, Nature.

[27]  Connelly Barnes,et al.  Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses , 2017, ArXiv.

[28]  Bruno Galerne,et al.  Microtexture inpainting through Gaussian conditional simulation , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[29]  Edward H. Adelson,et al.  The Design and Use of Steerable Filters , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  Leon A. Gatys,et al.  Texture Synthesis Using Convolutional Neural Networks , 2015, NIPS.

[31]  C.-C. Jay Kuo,et al.  Texture analysis and classification with tree-structured wavelet transform , 1993, IEEE Trans. Image Process..

[32]  Rob Fergus,et al.  Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks , 2015, NIPS.

[33]  Gabriel Peyré,et al.  Texture Synthesis with Grouplets , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  B. Julesz,et al.  Visual discrimination of textures with identical third-order statistics , 1978, Biological Cybernetics.

[35]  Marc Levoy,et al.  Fast texture synthesis using tree-structured vector quantization , 2000, SIGGRAPH.

[36]  Michael Ashikhmin,et al.  Synthesizing natural textures , 2001, I3D '01.

[37]  Lionel Moisan,et al.  A compact representation of random phase and Gaussian textures , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[38]  Dani Lischinski,et al.  Self Tuning Texture Optimization , 2015, Comput. Graph. Forum.

[39]  Bruno Galerne,et al.  Efros and Freeman Image Quilting Algorithm for Texture Synthesis , 2017, Image Process. Line.

[40]  Roland Vollgraf,et al.  Learning Texture Manifolds with the Periodic Spatial GAN , 2017, ICML.

[41]  B Julesz,et al.  Experiments in the visual perception of texture. , 1975, Scientific American.

[42]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[43]  Yann Gousseau,et al.  Exemplar-based Texture Synthesis: the Efros-Leung Algorithm , 2013, Image Process. Line.

[44]  Baining Guo,et al.  Real-time texture synthesis by patch-based sampling , 2001, TOGS.

[45]  Baining Guo,et al.  Chaos Mosaic: Fast and Memory Efficient Texture Synthesis , 2000 .

[46]  Bruno Galerne,et al.  Random Phase Textures: Theory and Synthesis , 2011, IEEE Transactions on Image Processing.

[47]  Dani Lischinski,et al.  Layered shape synthesis: automatic generation of control maps for non-stationary textures , 2009, SIGGRAPH 2009.

[48]  Song-Chun Zhu,et al.  What are Textons? , 2005, International Journal of Computer Vision.

[49]  Jean-Michel Morel,et al.  A Conditional Multiscale Locally Gaussian Texture Synthesis Algorithm , 2016, Journal of Mathematical Imaging and Vision.

[50]  Yann Gousseau,et al.  The dead leaves model: a general tessellation modeling occlusion , 2006, Advances in Applied Probability.

[51]  D. Mumford,et al.  Pattern Theory: The Stochastic Analysis of Real-World Signals , 2010 .

[52]  Sylvain Lefebvre,et al.  State of the Art in Example-based Texture Synthesis , 2009, Eurographics.

[53]  P Perona,et al.  Preattentive texture discrimination with early vision mechanisms. , 1990, Journal of the Optical Society of America. A, Optics and image science.

[54]  B. S. Manjunath,et al.  Texture Features for Browsing and Retrieval of Image Data , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[55]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[56]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[57]  Roland Vollgraf,et al.  Texture Synthesis with Spatial Generative Adversarial Networks , 2016, ArXiv.

[58]  Xi Chen,et al.  PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications , 2017, ICLR.

[59]  M. R. Turner,et al.  Texture discrimination by Gabor functions , 1986, Biological Cybernetics.

[60]  Nipun Kwatra,et al.  Texture optimization for example-based synthesis , 2005, ACM Trans. Graph..

[61]  Ken Perlin,et al.  An image synthesizer , 1988 .

[62]  Hugues Hoppe,et al.  Parallel controllable texture synthesis , 2005, SIGGRAPH 2005.

[63]  Thomas Brox,et al.  Generating Images with Perceptual Similarity Metrics based on Deep Networks , 2016, NIPS.

[64]  Koray Kavukcuoglu,et al.  Pixel Recurrent Neural Networks , 2016, ICML.

[65]  Jian Fan,et al.  Texture Classification by Wavelet Packet Signatures , 1993, MVA.

[66]  B. Julesz Dialogues on Perception , 1994 .

[67]  Julien Rabin,et al.  Wasserstein Barycenter and Its Application to Texture Mixing , 2011, SSVM.

[68]  Song-Chun Zhu,et al.  Exploring Texture Ensembles by Efficient Markov Chain Monte Carlo-Toward a 'Trichromacy' Theory of Texture , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[69]  Yang Lu,et al.  A Theory of Generative ConvNet , 2016, ICML.

[70]  J. van Wijk,et al.  Spot noise texture synthesis for data visualization , 1991, SIGGRAPH.

[71]  Song-Chun Zhu,et al.  Filters, Random Fields and Maximum Entropy (FRAME): Towards a Unified Theory for Texture Modeling , 1998, International Journal of Computer Vision.

[72]  Gang Liu,et al.  Texture synthesis through convolutional neural networks and spectrum constraints , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[73]  Bruno Galerne,et al.  Texture Inpainting Using Efficient Gaussian Conditional Simulation , 2017, SIAM J. Imaging Sci..

[74]  Hui Jiang,et al.  Generating images with recurrent adversarial networks , 2016, ArXiv.

[75]  P. Bickel,et al.  Texture synthesis and nonparametric resampling of random fields , 2006, math/0611258.

[76]  Jean Serra,et al.  Image Analysis and Mathematical Morphology , 1983 .

[77]  Roland Memisevic,et al.  Incorporating long-range consistency in CNN-based texture generation , 2016, ICLR.

[78]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[79]  Bruno Galerne,et al.  An Algorithm for Gaussian Texture Inpainting , 2017, Image Process. Line.

[80]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[81]  Abhinav Gupta,et al.  Generative Image Modeling Using Style and Structure Adversarial Networks , 2016, ECCV.

[82]  Jeremy S. De Bonet,et al.  Multiresolution sampling procedure for analysis and synthesis of texture images , 1997, SIGGRAPH.

[83]  B Julesz,et al.  Inability of Humans to Discriminate between Visual Textures That Agree in Second-Order Statistics—Revisited , 1973, Perception.

[84]  Gabriel Peyré,et al.  Sparse Modeling of Textures , 2009, Journal of Mathematical Imaging and Vision.

[85]  Song-Chun Zhu,et al.  Equivalence of Julesz Ensembles and FRAME Models , 2000, International Journal of Computer Vision.

[86]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[87]  Yang Lu,et al.  Learning FRAME Models Using CNN Filters , 2015, AAAI.

[88]  Bela Julesz,et al.  A theory of preattentive texture discrimination based on first-order statistics of textons , 2004, Biological Cybernetics.