3-D Structure from Visual Motion: Modeling, Representation and Observability

Abstract The problem of ‘structure from motion’ concerns the reconstruction of the three-dimensional structure of a scene from its projection onto a moving two-dimensional surface. Such a problem is solved effectively by the human visual system, judging from the ease with which we perform delicate control tasks involving vision as a sensor such as reaching for objects in the environment or driving a car. In this paper we study ‘structure from motion’ from the point of view of dynamical systems: we first formalize the problem of 3-D structure and motion reconstruction as the estimation of the state of certain nonlinear dynamical models. Then we study the feasibility of ‘structure from motion’ by analyzing the observability of such models. The models that define the visual motion estimation problem for feature points in the Euclidean 3-D space are not locally observable; however, the non-observable manifold can be easily isolated by imposing metric constraints on the state space. One of the peculiarities of vision as a sensor is its richness , which can be a disadvantage when we are interested only in few of the unknown parameters. For instance, if we want to control the direction of heading of our car by measuring brightness values on our retina, we have to overcome the effects that the shape of the environment, its reflectance properties, illumination and other quantities have on our measurements. Invariance to undesired parameters can be achieved by appropriate modeling or by choice of representation of the parameter space. We propose and analyze models for 3-D structure that are independent of 3-D motion and vice versa. Estimating unknown parameters from such models amounts to the identification of nonlinear and implicit systems with parameters on differentiable manifolds, such as a sphere or the so-called essential manifold . © 1997 Elsevier Science Ltd.