The computation of optical flow

Two-dimensional image motion is the projection of the three-dimensional motion of objects, relative to a visual sensor, onto its image plane. Sequences of time-orderedimages allow the estimation of projected two-dimensional image motion as either instantaneous image velocities or discrete image displacements. These are usually called the optical flow field or the image velocity field. Provided that optical flow is a reliable approximation to two-dimensional image motion, it may then be used to recover the three-dimensional motion of the visual sensor (to within a scale factor) and the three-dimensional surface structure (shape or relative depth) through assumptions concerning the structure of the optical flow field, the three-dimensional environment, and the motion of the sensor. Optical flow may also be used to perform motion detection, object segmentation, time-to-collision and focus of expansion calculations, motion compensated encoding, and stereo disparity measurement. We investigate the computation of optical flow in this survey: widely known methods for estimating optical flow are classified and examined by scrutinizing the hypothesis and assumptions they use. The survey concludes with a discussion of current research issues.

[1]  Robert J. Woodham.  Multiple light source optical flow , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[2]  Allen M. Waxman,et al.  Convected activation profiles and the measurement of visual motion , 1988, Proceedings CVPR '88: The Computer Society Conference on Computer Vision and Pattern Recognition.

[3]  Giulio Sandini,et al.  Estimation of depth from motion using an anthropomorphic visual sensor , 1990, Image Vis. Comput..

[4]  Alan L. Yuille,et al.  A Model for the Estimate of Local Velocity , 1990, ECCV.

[5]  W. F. Ranson,et al.  Determination of displacements using an improved digital correlation method , 1983, Image Vis. Comput..

[6]  S. BeaucheminDept.  The Structure of Occlusion in Fourier Space , 1995 .

[7]  H C Longuet-Higgins,et al.  The visual ambiguity of a moving plane , 1984, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[8]  Alessandro Verri,et al.  Identifying multiple motions from optical flow , 1992, ECCV.

[9]  T. Sato,et al.  Motion-detection model with two stages: Spatiotemporal filtering and feature matching , 1992 .

[10]  John K. Tsotsos,et al.  Techniques for disparity measurement , 1991, CVGIP Image Underst..

[11]  E. Delp,et al.  Estimating displacement vectors from an image sequence , 1989 .

[12]  Juyang Weng,et al.  A theory of image matching , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[13]  Håkan Bårman,et al.  Hierarchical curvature estimation in computer vision , 1991 .

[14]  A J Ahumada,et al.  Model of human visual-motion sensing. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[15]  James J. Little.  Accurate Early Detection of Discontinuities , 1992 .

[16]  Wilfried Enkelmann,et al.  Obstacle Detecion by Evaluation of Optical Flow Fields from Image Sequences , 1990, ECCV.

[17]  P. Anandan,et al.  Hierarchical Model-Based Motion Estimation , 1992, ECCV.

[18]  Brian G. Schunck,et al.  Image Flow Segmentation and Estimation by Constraint Line Clustering , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  B. D. Lucas.  Generalized image matching by the method of differences , 1985 .

[20]  Michael J. Black,et al.  Estimating Optical Flow in Segmented Images Using Variable-Order Parametric Models With Local Deformations , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Andrea Giachetti,et al.  The use of optical flow for the autonomous navigation , 1994, ECCV.

[22]  Valdis Berzins,et al.  Dynamic Occlusion Analysis in Optical Flow Fields , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Rama Chellappa,et al.  Time-to-X: analysis of motion through temporal parameters , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Ian Overington.  Gradient-Based Flow Segmentation and Location of the Focus of Expansion , 1987, Alvey Vision Conference.

[25]  A. Yuille,et al.  A model for the estimate of local image velocity by cells in the visual cortex , 1990, Proceedings of the Royal Society of London. B. Biological Sciences.

[26]  G. Granlund,et al.  Estimation of velocity, acceleration and disparity in time sequences , 1991, Proceedings of the IEEE Workshop on Visual Motion.

[27]  M. Shizawa,et al.  Principle of superposition: a common computational framework for analysis of multiple motion , 1991, Proceedings of the IEEE Workshop on Visual Motion.

[28]  Ajit Singh,et al.  Incremental estimation of image-flow using a Kalman filter , 1991, Proceedings of the IEEE Workshop on Visual Motion.

[29]  Edward M. Riseman,et al.  A data set for quantitative motion analysis , 1989, Proceedings CVPR '89: IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[30]  John L. Barron,et al.  Recursive estimation of time-varying motion and structure parameters , 1996, Pattern Recognit..

[31]  共立出版株式会社.  コンピュータ・サイエンス : ACM computing surveys , 1978 .

[32]  Yuan-Fang Wang,et al.  Experiments in computing optical flow with the gradient-based, multiconstraint method , 1987, Pattern Recognit..

[33]  Michael J. Black,et al.  Mixture models for optical flow computation , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[34]  G. L. Scott.  "Four-line" method of locally estimating optic flow , 1987, Image Vis. Comput..

[35]  E. Hildreth.  The computation of the velocity field , 1984, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[36]  A. Pentland,et al.  Robust estimation of a multi-layered motion representation , 1991, Proceedings of the IEEE Workshop on Visual Motion.

[37]  Ajit Singh,et al.  Incremental estimation of image flow using a Kalman filter , 1992, J. Vis. Commun. Image Represent..

[38]  Hans-Hellmut Nagel,et al.  Extending the 'Oriented Smoothness Constraint' into the Temporal Domain and the Estimation of Derivatives of Optical Flow , 1990, ECCV.

[39]  Nicola Ancona.  A Fast Obstacle Detection Method based on Optical Flow , 1992, ECCV.

[40]  James A. Storer,et al.  A split-merge parallel block-matching algorithm for video displacement estimation , 1992, Data Compression Conference, 1992..

[41]  W. Reichardt,et al.  Movement Detectors of the Correlation Type Provide Sufficient Information for Local Computation of the 2-D Velocity Field , 1988 .

[42]  K. Hanna.  Direct multi-resolution estimation of ego-motion and structure from motion , 1991, Proceedings of the IEEE Workshop on Visual Motion.

[43]  Keith J. Hanna,et al.  Combining stereo and motion analysis for direct estimation of scene structure , 1993, 1993 (4th) International Conference on Computer Vision.

[44]  Giulio Sandini,et al.  On The Estimation Of Depth From Motion Using An Anthropomorphic Visual Sensor , 1990, ECCV.

[45]  Vincent Torre,et al.  The Accuracy of the Computation of Optical Flow and of the Recovery of Motion Parameters , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  Hans-Hellmut Nagel,et al.  Displacement vectors derived from second-order intensity variations in image sequences , 1983, Comput. Vis. Graph. Image Process..

[47]  Gilad Adiv,et al.  Determining Three-Dimensional Motion and Structure from Optical Flow Generated by Several Moving Objects , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Yiannis Aloimonos,et al.  Determining the 3-D motion of a rigid surface patch without correspondence, under perspective projection: I. planar surfaces. II. curved surfaces , 1986, AAAI 1986.

[49]  Vishal Markandey,et al.  Multispectral constraints for optical flow computation , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[50]  V. Torre,et al.  The Use of Optical Ow for the Autonomous Navigation , 1994 .

[51]  Shahriar Negahdaripour,et al.  A generalized brightness change model for computing optical flow , 1993, 1993 (4th) International Conference on Computer Vision.

[52]  John A. Perrone,et al.  Simple technique for optical flow estimation , 1990 .

[53]  Michael J. Black,et al.  Robust dynamic motion estimation over time , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[54]  David J. Fleet.  Measurement of image velocity , 1992 .

[55]  K. Prazdny,et al.  Motion and Structure from Optical Flow , 1979, IJCAI.

[56]  Shahriar Negahdaripour,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence , 2004 .

[57]  Michael J. Black,et al.  A model for the detection of motion over time , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[58]  W. C. Karl,et al.  Eecient Multiscale Regularization with Applications to the Computation of Optical Flow 1 , 1993 .

[59]  Keith Langley,et al.  Recursive Filters for Optical Flow , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[60]  R. Haralick,et al.  The Facet Approach to Optic Flow , 1983 .

[61]  Alessandro Verri,et al.  Computing optical flow from an overconstrained system of linear algebraic equations , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[62]  Keith Langley,et al.  Computational analysis of non-Fourier motion , 1994, Vision Research.

[63]  Edward H. Adelson,et al.  Probability distributions of optical flow , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[64]  Edward H. Adelson,et al.  Representing moving images with layers , 1994, IEEE Trans. Image Process..

[65]  Steven D. Blostein,et al.  An error-weighted regularization algorithm for image motion-field estimation , 1993, IEEE Trans. Image Process..

[66]  T Poggio,et al.  Parallel integration of vision modules. , 1988, Science.

[67]  Ramesh C. Jain,et al.  Direct Computation of the Focus of Expansion , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[68]  Janet Aisbett,et al.  Optical Flow with an Intensity-Weighted Smoothing , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[69]  Christof Koch,et al.  Computing optical flow in resistive networks and in the primate visual system , 1989, [1989] Proceedings. Workshop on Visual Motion.

[70]  Anselm Spoerri,et al.  The early detection of motion boundaries , 1990, ICCV 1987.

[71]  Shaogang Gong,et al.  Parallel Computation Of Optic Flow , 1990, ECCV.

[72]  Naoki Mukawa.  Estimation of shape, reflection coefficients and illuminant direction from image sequences , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[73]  W. Clem Karl,et al.  Efficient multiscale regularization with applications to the computation of optical flow , 1994, IEEE Trans. Image Process..

[74]  Hans-Hellmut Nagel,et al.  On the Estimation of Optical Flow: Relations between Different Approaches and Some New Results , 1987, Artif. Intell..

[75]  D Marr,et al.  Theory of edge detection , 1979, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[76]  Joseph K. Kearney,et al.  Optical Flow Estimation: An Error Analysis of Gradient-Based Methods with Local Optimization , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[77]  Yee-Hong Yang,et al.  Experimental evaluation of motion constraint equations , 1991, CVGIP Image Underst..

[78]  Leif Haglund,et al.  Adaptive Multidimensional Filtering , 1991 .

[79]  H. C. Longuet-Higgins,et al.  A computer algorithm for reconstructing a scene from two projections , 1981, Nature.

[80]  Michal Irani,et al.  Recovery of ego-motion using image stabilization , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[81]  Ajit Singh,et al.  An estimation-theoretic framework for image-flow computation , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[82]  J. Heel.  Direct Estimation of Structure and Motion from Multiple Frames , 1990 .

[83]  J. Barron,et al.  Optical flow to measure minute increments in plant growth , 1994 .

[84]  F. W. Mounts.  A video encoding system with conditional picture-element replenishment , 1969 .

[85]  A. Waxman,et al.  On the Uniqueness of Image Flow Solutions for Planar Surfaces in Motion , 1985 .

[86]  John L. Barron,et al.  Image Reconstruction Error for Optical Flow , 1995, Research in Computer and Robot Vision.

[87]  Edward H. Adelson,et al.  The Laplacian Pyramid as a Compact Image Code , 1983, IEEE Trans. Commun..

[88]  Wilfried Enkelmann,et al.  Obstacle detection by evaluation of optical flow fields from image sequences , 1990, Image Vis. Comput..

[89]  Takeo Kanade,et al.  Adapting optical-flow to measure object motion in reflectance and x-ray image sequences (abstract only) , 1984, COMG.

[90]  T. M. Chin,et al.  Probabilistic and sequential computation of optical flow using temporal coherence , 1994, IEEE Trans. Image Process..

[91]  Hilary Buxton,et al.  Computation of optic flow from the motion of edge features in image sequences , 1984, Image Vis. Comput..

[92]  P. Pirsch,et al.  Advances in picture coding , 1985, Proceedings of the IEEE.

[93]  J. Hay,et al.  Optical motions and space perception: an extension of Gibson's analysis. , 1966, Psychological review.

[94]  HANS-HELLMUT NAGEL,et al.  On a Constraint Equation for the Estimation of Displacement Rates in Image Sequences , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[95]  Ramesh C. Jain,et al.  Segmentation of Frame Sequences Obtained by a Moving Observer , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[96]  Reinhard Klette,et al.  Evaluation of Differential Methods for Image Velocity Measurement , 1996, Comput. Artif. Intell..

[97]  Rama Chellappa,et al.  Automatic feature point extraction and tracking in image sequences for unknown camera motion , 1993, 1993 (4th) International Conference on Computer Vision.

[98]  J. H. Duncan,et al.  On the Detection of Motion and the Computation of Optical Flow , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[99]  D Regan,et al.  How do we avoid confounding the direction we are looking and the direction we are moving? , 1982, Science.

[100]  Rama Chellappa,et al.  A reliable optical flow algorithm using 3-D hermite polynomials , 1993 .

[101]  H. C. Longuet-Higgins,et al.  The interpretation of a moving retinal image , 1980, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[102]  Libor Spacek,et al.  Edge detection and motion detection , 1986, Image Vis. Comput..

[103]  Shmuel Peleg,et al.  A Three-Frame Algorithm for Estimating Two-Component Image Motion , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[104]  James J. Little,et al.  Parallel Optical Flow Using Local Voting , 1988, [1988 Proceedings] Second International Conference on Computer Vision.

[105]  P. Burt,et al.  Mechanisms for isolating component patterns in the sequential analysis of multiple motion , 1991, Proceedings of the IEEE Workshop on Visual Motion.

[106]  David J. Fleet,et al.  Performance of optical flow techniques , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[107]  Muralidhara Subbarao,et al.  Bounds on time-to-collision and rotational component from first-order derivatives of image flow , 1990, Comput. Vis. Graph. Image Process..

[108]  Jitendra Malik,et al.  Robust computation of optical flow in a multi-scale differential framework , 1993, 1993 (4th) International Conference on Computer Vision.

[109]  Thomas S. Huang,et al.  Uniqueness and Estimation of Three-Dimensional Motion Parameters of Rigid Objects with Curved Surfaces , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[110]  Jake K. Aggarwal,et al.  On the computation of motion from sequences of images-A review , 1988, Proc. IEEE.

[111]  J. Barron,et al.  Motion and Structure from Time-varying Optical Flow , 1995 .

[112]  Alexander A. Sawchuk,et al.  A region matching motion estimation algorithm , 1991, CVGIP Image Underst..

[113]  Jacques Droulez,et al.  Stereo Correspondence From Optic Flow , 1990, ECCV.

[114]  Venkataraman Sundareswaran.  A Fast Method to Estimate Sensor Translation , 1992, ECCV.

[115]  Thomas S. Huang,et al.  Estimating three-dimensional motion parameters of a rigid planar patch, II: Singular value decomposition , 1982 .

[116]  S. Ullman,et al.  The interpretation of visual motion , 1977 .

[117]  J. Kittler,et al.  Robust motion analysis , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[118]  Heinrich Niemann,et al.  Finite element method for determination of optical flow , 1992, Pattern Recognit. Lett..

[119]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[120]  Allen M. Waxman,et al.  Contour Evolution, Neighborhood Deformation, and Global Image Flow: Planar Surfaces in Motion , 1985 .

[121]  E H Adelson,et al.  Spatiotemporal energy models for the perception of motion. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[122]  Jerry L. Prince,et al.  Motion estimation from tagged MR image sequences , 1992, IEEE Trans. Medical Imaging.

[123]  William B. Thompson,et al.  Disparity Analysis of Images , 1980, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[124]  Gary Whitten.  A framework for adaptive scale space tracking solutions to problems in computational vision , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[125]  Christof Koch,et al.  An adaptive multi-scale approach for estimating optical flow: computational theory and physiological implementation , 1991, Proceedings of the IEEE Workshop on Visual Motion.

[126]  Bernd Jähne.  Motion Determination in Space-Time Images , 1990, ECCV.

[127]  Hans-Hellmut Nagel,et al.  An Investigation of Smoothness Constraints for the Estimation of Displacement Vector Fields from Image Sequences , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[128]  E. Dubois,et al.  The sampling and reconstruction of time-varying imagery with application in video systems , 1985, Proceedings of the IEEE.

[129]  Keith Langley,et al.  Vertical and horizontal disparities from phase , 1991, Image Vis. Comput..

[130]  David W. Murray,et al.  Scene Segmentation from Visual Motion Using Global Optimization , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[131]  Takeo Kanade,et al.  A locally adaptive window for signal matching , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[132]  C.-C. Jay Kuo,et al.  Computation of Dense Optical Flow with a Parametric Smoothness Model , 1993, J. Vis. Commun. Image Represent..

[133]  Michael J. Black.  Robust incremental optical flow , 1992 .

[134]  J. D. Robbins,et al.  Motion-compensated television coding: Part I , 1979, The Bell System Technical Journal.

[135]  Mandyam V. Srinivasan,et al.  Measurement of optical flow by a generalized gradient scheme , 1991 .

[136]  G. Dullerud.  A Computational Framework , 1996 .

[137]  Wilfried Enkelmann,et al.  Investigations of multigrid algorithms for the estimation of optical flow fields in image sequences , 1988, Comput. Vis. Graph. Image Process..

[138]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[139]  H Zinner.  Determining the kinematic parameters of a moving imaging sensor by processing spatial and temporal intensity changes. , 1986, Journal of the Optical Society of America. A, Optics and image science.