Using object descriptions in a schema network for machine vision

Computer interpretation of a single static image of a typical natural scene requires the application of a large amount of detailed knowledge. This dissertation explores the information and control structures needed for knowledge-directed interpretation of natural outdoor scenes. A schema network represents object descriptions, relations among objects, and control knowledge. Each node of the network, a schema, contains both a declarative structure and references to one or more interpretation strategies. The declarative portion of the schema describes the composition of an object including the spatial relations of its parts and their possible appearances in an image. The interpretation strategies are object-specific procedures for creating hypotheses of the existence of the object; this procedural representation of control information provides a natural form for expressing the dynamic nature of the image interpretation process. A schema instance is created when a schema is activated either by a top-down request for a goal or by bottom-up detection of key events in the image. Schema instances continually interact with one another, either through a channel set up when a goal is requested or through hypotheses created in a blackboard data structure. Several schema instances can work simultaneously on relatively independent portions of the interpretation, thus exploiting the potential for parallelism. By selectively grouping line and region primitives into descriptions of parts of a scene, the cooperative activities of the schema instances construct the final interpretation network. The system was tested on six images from four scenes. The parallel execution of the interpretation strategies is simulated and experimental traces are included to illustrate their overlapping activity. The resulting interpretations contain both the association between object structures and image events, as well as three-dimensional descriptions of some of the objects in the scenes.