EYE MOVEMENTS IN NATURAL ENVIRONMENTS

How do the limitations of attention and working memory constrain acquisition of information in the context of natural behavior? Overt fixations carry much information about current attentional state, and are a revealing indicator of this process. Fixation patterns in natural behavior are largely determined by the momentary task. The implication of this is that fixation patterns are a learnt behavior. We review several recent findings that reveal some aspects of this learning. In particular, subjects learn the structure and dynamic properties of the world in order to fixate critical regions at the right time. They also learn how to allocate attention and gaze to satisfy competing demands in an optimal fashion, and are sensitive to changes in those demands. Understanding exactly how tasks exert their control on gaze is a critical issue for future research. Elsevier AMS Ch30-I044980 Job code: EMAW 14-2-2007 1:17p.m. Page:643 Trimsize:165×240MM Basal Fonts:Times Margins:Top:4.6pc Gutter:4.6pc Font Size:10/12 Text Width:30pc Depth:43 Lines Ch. 30: Learning Where to Look 643 A central feature of human cognition is the strict limitation on the ability to acquire visual information from the environment, set by limitations in attention. Related to this are the limits in retaining this information, set by the capacity of working memory. We are far from understanding how the organization of the brain leads to these limitations. We also have little understanding of how they influence the way that visual perception operates in the natural world, in the service of everyday visually guided behavior. Consideration of how the limited processing capacity of cognition influences acquisition of visual information leads us to the problem of how such acquisition is controlled. It is not really possible to address the question of precisely what information is selected from the image, and when it is selected, in the context of traditional experimental paradigms, where the trial structure is designed to measure a particular visual operation over repeated instances, each of short duration. In natural behavior, on the other hand, observers control what information is selected from the image and when it is selected. By observing natural behavior, knowledge of the task structure often allows quite well constrained inferences about the underlying visual computations, on a time scale of a few hundred milliseconds. 1. Eye movements and task structure How can we study the acquisition of information in the natural world? Although incomplete, eye movements are an overt manifestation of the momentary deployment of attention in a scene. Covert attentional processes, of course, mean that other information is processed as well, but overt fixations carry a tremendous amount of information about current attentional state, and provide an entree to studying the problem (Findlay & Gilchrist, 2003). Investigation of visual performance in natural tasks is now much more feasible, given the technical developments in monitoring eye, head, and hand movements in unconstrained observers, as well as the development of complex virtual environments. This allows some degree of experimental control while allowing relatively natural behavior. In natural behavior, the task structure is evident, and this allows the role of individual fixations to be fairly easily interpreted, because the task provides an external referent for the internal computations. In contrast, when subjects simply passively view images, the experimenter often has little control of, and no access to, what the observer is doing. When viewing pictures, observers may be engaged in object recognition, remembering object locations and identity, or performing some other visual operation. Immersion in a real scene probably calls for different kinds of visual computations, because observers may be interacting with the objects in the scene. When viewing images of scenes, some regularities in fixation patterns can be explained by image properties such as contrast or chromatic salience. However, these factors usually account for only a modest proportion of the variance (Itti & Koch, 2001; Mannan, Ruddock & Wooding, 1997; Parkhurst, Law, & Neibur, 2002). Over the past ten years, a substantial amount of evidence has accumulated about deployment of gaze during ongoing natural behavior. In extended visuomotor tasks such as driving, walking, sports, playing a piano, hand-washing, and making tea or sandwiches, Elsevier AMS Ch30-I044980 Job code: EMAW 14-2-2007 1:17p.m. Page:644 Trimsize:165×240MM Basal Fonts:Times Margins:Top:4.6pc Gutter:4.6pc Font Size:10/12 Text Width:30pc Depth:43 Lines 644 M. M. Hayhoe et al. the central finding is that fixations are tightly linked to the performance of the task (Hayhoe, Shrivastrava, Mruczek & Pelz, 2003; Land & Furneaux, 1997; Land & Lee, 1994; Land, Mennie, & Rusted, 1999; Patla & Vickers, 1997; Pelz & Canosa, 2001; Turano, Geruschat, & Baker, 2003). Subjects exhibit regular, often quite stereotyped fixation sequences as they step through the task. Very few irrelevant areas are fixated. Figure 1 shows an example of the clustering of fixations on task-specific regions when a subject makes a sandwich. This is hard to capture in a still image, but can be clearly appreciated in video sequences such as those in Hayhoe et al. (2003). A feature of the relationship of the fixations to the task is that they are tightly linked, in time, to the actions (Land et al., 1999; Hayhoe et al., 2003). The temporal linkage has been demonstrated clearly by Johansson, Westling, Backstrom, & Flanagan (2001), who measured fixation locations and hand path while a subject picked up a bar and maneuvered the tip past an obstacle, to contact a switch. Fixations were made at critical points such as the tip of the obstacle while the bar was moved around it, and then on the switch once the bar had cleared the obstacle. Gaze arrived at the critical point just before the action, and departed just as the action was accomplished. This is illustrated in Figure 2. This aspect of natural behavior, where observers acquire the specific information they need just at the point it is required in the task, was called a “just-in-time” strategy (Ballard, Hayhoe, & Pelz, 1995). In their experiment, subjects copied a pattern of colored blocks (the Model) using pieces in a Resource area, which they picked up and placed in Figure 1. Fixations made by an observer while making a peanut butter and jelly sandwich, indicated by yellow circles. Images were taken from a camera mounted on the head, and a composite image mosaic was formed by integrating over different head positions using a method described in Rothkopf and Pelz (2004) et al. (The reconstructed panorama shows artifacts because the translational motion of the subject was not taken into account.) Fixations are shown as yellow circles, with a diameter proportional to fixation duration. The red lines indicate the saccades. Note that almost all fixations fall on task relevant objects. (See Color Plate 10.) Elsevier AMS Ch30-I044980 Job code: EMAW 14-2-2007 1:17p.m. Page:645 Trimsize:165×240MM Basal Fonts:Times Margins:Top:4.6pc Gutter:4.6pc Font Size:10/12 Text Width:30pc Depth:43 Lines Ch. 30: Learning Where to Look 645 Time relative to kinematic events C um ul at iv e fr eq ue nc y