Robust computation of optical flow in a multi-scale differential framework

We have developed a new algorithm for computing optical flow in a differential framework. The image sequence is first convolved with a set of linear, separable spatiotemporal filter kernels similar to those that have been used in other early vision problems such as texture and stereopsis. The brightness constancy constraint can then be applied to each of the resulting images, giving us, in general, an overdetermined system of equations for the optical flow at each pixel. There are three principal sources of error: (a) stochastic error due to sensor noise (b) systematic errors in the presence of large displacements and (c) errors due to failure of the brightness constancy model. Our analysis of these errors leads us to develop an algorithm based on a robust version of total least squares. Each optical flow vector computed has an associated reliability measure which can be used in subsequent processing. The performance of the algorithm on the data set used by Barron et al. (IJCV 1994) compares favorably with other techniques. In addition to being separable, the filters used are also causal, incorporating only past time frames. The algorithm is fully parallel and has been implemented on a multiple processor machine.

[1]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[2]  A. Madansky.  The fitting of straight lines when both variables are subject to error , 1959 .

[3]  P. Sprent.  Models in regression and related topics , 1971 .

[4]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[5]  Peter E. Hart,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[6]  S. Addelman,et al.  Fitting straight lines when both variables are subject to error. , 1978, Life sciences.

[7]  Claude L. Fennema,et al.  Velocity determination in scenes containing several moving objects , 1979 .

[8]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[9]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[10]  R. Young.  GAUSSIAN DERIVATIVE THEORY OF SPATIAL VISION: ANALYSIS OF CORTICAL CELL RECEPTIVE FIELD LINE-WEIGHTING PROFILES. , 1985 .

[11]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Hans-Hellmut Nagel,et al.  On the Estimation of Optical Flow: Relations between Different Approaches and Some New Results , 1987, Artif. Intell..

[13]  E. Adelson,et al.  Early vision and texture perception , 1988, Nature.

[14]  Josef Kittler,et al.  Proceedings of the 4th International Conference on Pattern Recognition , 1988 .

[15]  P Perona,et al.  Preattentive texture discrimination with early vision mechanisms. , 1990, Journal of the Optical Society of America. A, Optics and image science.

[16]  Kenji Mase,et al.  Simultaneous multiple optical flow estimation , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[17]  O. Braddick,et al.  Direction discrimination for band-pass filtered random dot kinematograms , 1990, Vision Research.

[18]  Alessandro Verri,et al.  Computing optical flow from an overconstrained system of linear algebraic equations , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[19]  A. Verri,et al.  Differential techniques for optical flow , 1990 .

[20]  Sabine Van Huffel,et al.  Total least squares problem - computational aspects and analysis , 1991, Frontiers in applied mathematics.

[21]  Edward H. Adelson,et al.  Probability distributions of optical flow , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[22]  Jitendra Malik,et al.  Determining Three-Dimensional Shape from Orientation and Spatial Frequency Disparities , 1991, ECCV.

[23]  Jitendra Malik,et al.  A Computational Framework for Determining Stereo Correspondence from a Set of Linear Spatial Filters , 1991, ECCV.

[24]  Vishal Markandey,et al.  Total least squares fitting spatiotemporal derivatives to smooth optical flow fields , 1992, Defense, Security, and Sensing.

[25]  Jitendra Malik,et al.  Computational framework for determining stereo correspondence from a set of linear spatial filters , 1992, Image Vis. Comput..

[26]  Jitendra Malik,et al.  Robust computation of optical flow in a multi-scale differential framework , 1993, 1993 (4th) International Conference on Computer Vision.

[27]  Michael J. Black,et al.  A framework for the robust estimation of optical flow , 1993, 1993 (4th) International Conference on Computer Vision.

[28]  David J. Fleet,et al.  Computation of component image velocity from local phase information , 1990, International Journal of Computer Vision.

[29]  A. Verri,et al.  A computational approach to motion perception , 1988, Biological Cybernetics.

[30]  M. R. Turner,et al.  Texture discrimination by Gabor functions , 1986, Biological Cybernetics.

[31]  M. Srinivasan.  Generalized gradient schemes for the measurement of two-dimensional image motion , 1990, Biological Cybernetics.

[32]  B. Ripley,et al.  Robust Statistics , 2018, Wiley Series in Probability and Statistics.

[33]  David J. Heeger,et al.  Optical flow using spatiotemporal filters , 2004, International Journal of Computer Vision.

[34]  Roberto Battiti,et al.  Computing optical flow across multiple scales: An adaptive coarse-to-fine strategy , 1991, International Journal of Computer Vision.

[35]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[36]  Gene H. Golub,et al.  An analysis of the total least squares problem , 1980, Milestones in Matrix Computation.