Efficient Coding of Visual Scenes by Grouping and Segmentation: Theoretical Predictions and Biological Evidence

Efficient coding by scene segmentation Lee & Yuille Efficient coding of visual scenes by grouping and segmentation: theoretical predictions and biological evidence Tai Sing Lee & Alan L. Yuille Introduction The goal of this chapter is to present computational theories of scene coding by image segmentation and to suggest their relevance for understanding visual cortical function and mechanisms. We will first introduce computational theories of image and scene segmentation and show their relationship to efficient encoding. Then we discuss and evaluate the rel- evant physiological data in the context of these computational frameworks. It is hoped that this will stimulate quantitative neurophysiological investigations of scene segmentation guided by computational theories. Our central conjecture is that areas V1 and V2, in addition to encoding fine details of images in terms of filter responses, compute a segmentation of images which allow a more compact and parsimonious encoding of images in terms of the properties of regions and surfaces in the visual scene. This conjecture is based on the observation that neurons and their retinotopic arrangement in these visual areas can represent information precisely, thus furnishing an appropriate computational and representational infrastructure for this task. Segmentation detects and extracts coherent regions in an image and then encode the image in terms of probabilistic models of surfaces and regions in it, in the spirit of Shannon’s theory of information. This representation facilitates visual reasoning at the level of regions and their boundaries, without worrying too much about all the small details in the image. Figure 1 gives three examples which illustrate the meaning of higher level efficient encoding of scenes. Firstly, consider Kanizsa’s (1979) famous illusory triangle (Figure 1a). It is simpler to explain it as a white triangle in front of, and partially occluding, three black circular discs rather than as three pac-mens which are accidentally aligned to each other. Indeed this simple explanation is what human perceive and, in fact, the perception of a triangle is so strong that we hallucinate the surface of the triangle as being brighter than the background and perceive sharp boundaries to the triangle even at places where there is no direct visual cues. Secondly, when confronted with the image shown in Figure 1b (Ramachandran 1988), we perceive it as a group of convex spheres mixed together with a group of concave indentations (e.g. an egg carton partly filled with eggs). This interpretation is more parsimonious than describing every detail of the intensity shading and other image features. Thirdly, at first glance, the image in Figure 1c (Gregory 1970) appears to be a collection of random dots and hence would not have a simple encoding. But the encoding becomes greatly simplified once the viewer perceives the Dalmation dog and can invoke a dog model. The viewer will latch on to this interpretation whenever he sees it again, underscoring the powerful interaction between memory and perception when generating an efficient perceptual description. These three examples suggest that we can achieve a tremendous amount of data compression by interpreting images in terms of the structure of the scene. They suggest a succession of increasingly more compact and semantically more meaningful codes as we move up the visual hierarchy. These codes go beyond efficient coding of images based on Gabor wavelet responses (Daugman 1985, Lee 1996) or independent components (Olshausen and Field 1996, Bell and Sejnowski 1997, Lewicki and Olshausen 1999). In this chapter, we will concentrate on image segmentation which is the process that partitions an image into regions, producing a clear delineation of the boundaries between regions and the labelling of properties of the regions. The definition of “regions” is a flexible one. In this chapter, we focus on early visual processing and so a region is defined to be part of an image that is characterized by a set of (approximately) homogeneous visual cues, such as color, luminance, or texture. These regions can correspond to 3D surfaces in the visual scene, or they can be parts of a 3D surface defined by (approximately) constant texture, albedo, or color (e.g. the red letters “No Parking” on a white stop sign). Based on a single image, however, it is often difficult to distinguish between these two interpretations. At a higher level of vision, the definition of region is more complex and can involve hierarchical structures involving objects and scene structures. The approach we have taken stems from the following computational perspective about the function of the visual system. We hold it to be self-evident that the purpose of the visual system is to interpret input images in terms of objects MIT Press Page