Explanation-mediated vision: making sense of the world through causal analysis

A long-standing and widely acknowledged problem in artificial intelligence is the dissociation of perception and cognition. This dissociation has led, on one hand, to cognitive models that bear uncertain relation to the problems of purposefully interacting with the world, and on the other hand to perceptual algorithms that produce data whose meaning and relevance is usually unknown, if considered at all. In this thesis I argue that the key to integrating perception and cognition is the development of good domain models of the perceptual world, describing the causality of the physical world, how that causality is manifest in the activity of the senses, and how special aspects of that causality are privileged as "functional," or goal-related. To support this argument I describe a number of artificial vision systems in which "perceptual" and "cognitive" computations interact and constrain each other via causal-perceptual models. These systems produce meaningful explanations of the internal dynamics of scenes. These explanations support a variety of high-level intelligent behaviors, including planning, robotic manipulation, and design evaluation. In the course of scene analysis, these systems use functional and causal anomalies in ongoing explanations to generate visual queries, thus providing control of focus of attention in low-level visual processing. The causal models employed by these systems are relatively small, but sufficiently powerful to generate explanations of highly complex structures. Much of these models can be recycled between visually disparate domains, suggesting wide generality. Given the success of these models, I consider the prospects for a cognitive psychology of vision, in which the knowledge that drives human visual understanding can be captured, formalized, and deployed in machines.