Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks

It is a long standing question how biological systems transform visual inputs to robustly infer high-level visual information. Research in the last decades has established that much of the underlying computations take place in a hierarchical fashion along the ventral visual pathway. However, the exact processing stages along this hierarchy are difficult to characterise. Here we present a method to generate stimuli that will allow a principled description of the processing stages along the ventral stream. We introduce a new parametric texture model based on the powerful feature spaces of convolutional neural networks optimised for object recognition. We show that constraining a spatial summary statistic over feature maps suffices to synthesise high-quality natural textures. Moreover we establish that our texture representations continuously disentangle high-level visual information and demonstrate that the hierarchical parameterisation of the texture model naturally enables us to generate novel types of stimuli for systematically probing mid-level vision.