Texture modeling and synthesis using joint statistics of complex wavelet coefficients

We present a statistical characterization of texture images in the context of an overcomplete complex wavelet transform. The characterization is based on empirical observations of statistical regularities in such images, and parameterized by (1) the local autocorrelation of the coeÆcients in each subband; (2) both the local auto-correlation and cross-correlation of coeÆcient magnitudes at other orientations and spatial scales; and (3) the rst few moments of the image pixel histogram. We develop an eÆcient algorithm for synthesizing random images subject to these constraints using alternated projections, and demonstrate its e ectiveness on a wide range of synthetic and natural textures. In particular, we show that many important structural elements in textures (e.g., edges, repeated patterns or alternated patches of simpler texture), can be captured through joint second order statistics of the coeÆcient magnitudes. We also show the exibility of the representation, by applying to a variety of tasks which can be viewed as constrained image synthesis problems, such as spatial and spectral extrapolation. 1Instituto de Optica, CSIC, Serrano 121, 28006 Madrid, SPAIN. iodpm79@pinar2.csic.es 2Center for Neural Science, and Courant Inst. of Mathematical Sciences, New York University, New York, NY 10003. eero.simoncelli@nyu.edu JP is supported by a fellowship from the Consejo Superior de Investigaciones Cienti cas (CSIC), and the Comision Interministerial de Ciencia y Tecnologia (CICYT, Spain), under grant TIC97-325. EPS is supported by an Alfred P. Sloan Research Fellowship, NSF CAREER grant MIP-9796040, and the Sloan Center for Theoretical Neurobiology at NYU. Vision is arguably our most important sensory system, judging from both the ubiquity of visual forms of communication, and the large proportion of the human brain devoted to visual processing. Nevertheless, it has proven diÆcult to establish a good mathematical de nition (in the form of a statistical model) for visual images. The set of visual images is enormous, and yet only a small fraction of these are likely to be encountered in a natural setting [43, 25, 19, 54]. Thus, a statistical prior model, even one that partially captures these variations in likelihood, can substantially bene t image processing and arti cial vision systems. In addition, many authors have proposed that biological visual systems have evolved to take advantage of the statistical properties of the signals to which they are exposed [e.g., 3, 4, 2], thus suggesting a direct link between image statistics and visual processing. In order to characterize visual images statistically, one must make some sort of restriction on the probability model. The most common assumptions are locality (the characterization is speci ed on local spatial neighborhoods), stationarity (the statistics depend only on relative spatial position within the image), and a parametric form for the density (e.g., Gaussian). The subclass of images that we commonly call \visual texture" seems most consistent with local stationary density models. Traditional de nitions of texture can be classi ed into \structural" and \statistical" [34], the rst consisting of a set of repeated deterministic features, and the second corresponding to a sample drawn from a probability density. Such a distinction has turned out to be somewhat arti cial: For example, Zhu et al. [69] have demonstrated that is possible to capture and reproduce structural elements in texture using purely statistical models. Furthermore, many real-world textures seem to incorporate both aspects, in that they can be described as a set of repeating structural elements subject to some randomness in their location, size, color, orientation, etc. This observation leads us to seek a single method of representing textures. In this paper, we develop a fully statistical description, and demonstrate that it is also able to capture and reproduce a wide variety of structural elements. Julesz pioneered the statistical characterization of textures by hypothesizing that the Nth-order joint empirical densities (for some unspeci ed N) of neighborhoods of image pixels, could be used to partition textures into classes that are indistinguishable to a human observer [40]. This work thus established the use of both the locality and stationarity assumptions, the goal of determining a minimal set of statistical constraints, and the validation of texture models using human observers. Since then, researchers have explored a wide variety of approaches for texture characterization and synthesis. One of the most basic distinctions between the various approaches is the choice of representation. Starting with Julesz, many authors have worked directly on the statistical attributes of local spatial neighborhoods of pixels, typically in the form of a Markov random eld [e.g., 35, 42, 17]. But most others attempt to simplify the description of the density by rst processing with a set of linear lters, such as Gabor lters or a multiscale basis. The use of localized multi-scale multi-orientation sets of bandpass lters is inspired by what is known of biological visual processing, and justi ed by recent studies of the higher-order statistical properties of such representations (we discuss this further in section 1). These lters may be held xed, or chosen adaptively depending on the image statistics [e.g., 10, 26, 22, 59, 51]. Assuming a preconditioning subband decomposition, most texture characterizations