Dynamic estimation in computational vision

Spatial coherence constraints are commonly used to regularize the problems of reconstructing dense visual fields like depth, shape, and motion. Recent developments in theory and practice show that the local nature of spatial coherence constraints allows us to solve single-frame reconstruction problems efficiently with, for example, multiresolution approaches. While it is reasonable to impose temporal as well as spatial coherence on the unknown for a more robust estimation through data fusion over both space and time, such temporal, multi-frame extensions of the problems have not been as widely considered, perhaps due to the different and severe computational demands imposed by the sequential arrival of the image data. We present here an efficient filtering algorithm for sequential estimation of dense visual fields, using stochastic descriptor dynamic system models to capture temporal smoothness and dynamics of the fields. Theoretically, standard Kalman filtering techniques (generalized for stochastic descriptor systems) are applicable to solving temporally-extended visual field reconstruction problems, but their implementation is practically impossible because of the high dimensionality and because the time-varying nature of such problems requires on-line propagation of large covariance matrices. By exploiting the inherent local spatial structure of the reconstruction problem, however, we have developed filtering techniques that effectively approximate the information form of the Kalman filter. This is achieved by replacing covariance propagation steps with what are essentially low-order spatial model identification steps, in which spatial models with a strictly local support are constructed based on covariance information. In effect, we are decomposing the multi-frame problem into a series of Bayesian single-frame problems, in which the spatial prior model used reflects knowledge from the previous image frames. The resulting filtering algorithm has memory and computational requirements of O(N) each for a frame of data, where N is the number of pixels in a frame, and, additionally, the filter is implementable in parallel. As low-level visual field reconstruction is often considered to be a front-end in a hierachical visual processing system and thus might be VLSI-implemented, we have also designed a square root version of the information Kalman filter as an alternative algorithm with a reduced numerical dynamic range. The square root information 3 filter features an efficient, iterative computational structure and is parallelizable as well. Experiments have shown several beneficial effects of our multi-frame formulation applied to the sequential estimation of optical flow. For example, temporal assimilation of the data makes the reconstruction more robust to noise. Also, there are cases where the classic "aperture problem" of motion vision cannot be resolved satisfactorily by spatial regularization alone but is dramatically resolved by the additional temporal coherence constraint. Thesis Supervisor: Alan S. Willsky Title: Professor, Electrical Engineering