Recovering three-dimensional structure from motion with surface reconstruction

This paper addresses the computational role that the construction of a complete surface representation may play in the recovery of 3-D structure from motion. We first discuss the need to integrate surface reconstruction with the structure-from-motion process, both on computational and perceptual grounds. We then present a model that combines a feature-based structure-from-motion algorithm with a smooth surface interpolation mechanism. This model allows multiple surfaces to be represented in a given viewing direction, incorporates constraints on surface structure from object boundaries, and segregates image features onto multiple surfaces on the basis of their 2-D image motion. We present the results of computer simulations that relate the qualitative behavior of this model to psychophysical observations. In a companion paper, we discuss further perceptual observations regarding the possible role of surface reconstruction in the human recovery of 3-D structure from motion.

[1]  G J Andersen,et al.  Shape and depth perception from parallel projections of three-dimensional motion. , 1984, Journal of experimental psychology. Human perception and performance.

[2]  J. Todd.  Visual information about rigid and nonrigid motion: a geometric analysis. , 1982, Journal of experimental psychology. Human perception and performance.

[3]  B. Anderson,et al.  Optimal Filtering , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[4]  J S Lappin,et al.  Accurate visual measurement of three-dimensional moving patterns. , 1983, Science.

[5]  S. Ullman,et al.  Rigidity and Smoothness of Motion , 1987 .

[6]  Federico Girosi,et al.  Parallel and Deterministic Algorithms from MRFs: Surface Reconstruction , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Shimon Ullman,et al.  Computational Studies in the Interpretation of Structure and Motion: Summary and Extension , 1983 .

[8]  G. Johansson,et al.  Visual Perception of Bending Motion , 1973, Perception.

[9]  George A. Kaplan,et al.  Kinetic disruption of optical texture: The perception of depth at an edge , 1969 .

[10]  S Ullman,et al.  Maximizing Rigidity: The Incremental Recovery of 3-D Structure from Rigid and Nonrigid Motion , 1984, Perception.

[11]  G Sperling,et al.  Kinetic depth effect and identification of shape. , 1989, Journal of experimental psychology. Human perception and performance.

[12]  R. M. Siegel,et al.  Perception of three-dimensional structure from motion in monkey and man , 1988, Nature.

[13]  John R. Kender,et al.  Visual Surface Reconstruction Using Sparse Depth Data , 1986, CVPR 1986.

[14]  Alex P. Pentland,et al.  From Pixels to Predicates: Recent Advances in Computational and Robotic Vision , 1986, IEEE Expert.

[15]  E C Hildreth,et al.  Incremental rigidity scheme for recovering structure from motion: position-based versus velocity-based formulations. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[16]  J T Petersik,et al.  Three-dimensional object constancy: Coherence of a simulated rotating sphere in noise , 1979, Perception & psychophysics.

[17]  J T Todd,et al.  The perception of three-dimensional structure from rigid and nonrigid motion , 1984, Perception & psychophysics.

[18]  W. B. Thompson,et al.  Relative motion: Kinetic information for the order of depth at an edge , 1987, Perception & psychophysics.

[19]  J. Allman,et al.  Perceptions of Depth Elicited by Occluded and Shearing Motions of Random Dots , 1988, Perception.

[20]  M S Landy,et al.  Parallel model of the kinetic depth effect using local computations. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[21]  G. Edelman,et al.  Signal and sense : local and global order in perceptual maps , 1990 .

[22]  D. Scully.  Visual Perception of Biological Motion , 1987 .

[23]  H. Ando.  Dynamic reconstruction of 3D structure and 3D motion , 1991, Proceedings of the IEEE Workshop on Visual Motion.

[24]  J T Todd,et al.  Perception of structure from motion: is projective correspondence of moving elements a necessary condition? , 1985, Journal of experimental psychology. Human perception and performance.

[25]  T. Poggio,et al.  Visual Integration and Detection of Discontinuities: The Key Role of Intensity Edges , 1987 .

[26]  J. Aloimonos,et al.  Motion–boundary illusions and their regularization , 1990, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[27]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[28]  Donald D. Hoffman.  Inferring local surface orientation from motion fields. , 1982, Journal of the Optical Society of America.

[29]  G. Phillips,et al.  Cooperative phenomena in the perception of motion direction. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[30]  David W. Eby,et al.  Relative motion parallax and the perception of structure from motion , 1989, [1989] Proceedings. Workshop on Visual Motion.

[31]  Thomas S. Huang,et al.  Uniqueness and Estimation of Three-Dimensional Motion Parameters of Rigid Objects with Curved Surfaces , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Arthur Gelb,et al.  Applied Optimal Estimation , 1974 .

[33]  M. Sanders.  Handbook of Sensory Physiology , 1975 .

[34]  M. Landy,et al.  Kinetic depth effect and optic flow—I. 3D shape from Fourier motion , 1989, Vision Research.

[35]  H. C. Longuet-Higgins,et al.  The interpretation of a moving retinal image , 1980, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[36]  E. Polak.  Introduction to linear and nonlinear programming , 1973 .

[37]  S. McKee,et al.  Sequential recruitment in the discrimination of velocity. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[38]  Stefan Treue,et al.  Human perception of structure from motion , 1991, Vision Research.

[39]  S. Ullman,et al.  The interpretation of visual motion , 1977 .

[40]  Christopher M. Brown.  Advances in computer vision , 1987 .

[41]  G Sperling,et al.  Ratings of kinetic depth in multidot displays. , 1989, Journal of experimental psychology. Human perception and performance.

[42]  Andrew Blake,et al.  Visual Reconstruction , 1987, Deep Learning for EEG-Based Brain–Computer Interfaces.

[43]  G J Andersen,et al.  A Counterexample to the Rigidity Assumption in the Visual Perception of Structure from Motion , 1984, Perception.

[44]  Gunnar Johansson,et al.  Visual Event Perception , 1978 .

[45]  J J Koenderink,et al.  Depth and shape from differential perspective in the presence of bending deformations. , 1986, Journal of the Optical Society of America. A, Optics and image science.

[46]  J T Petersik,et al.  Recovery of structure from motion: Implications for a performance theory based on the structure-from-motion theorem , 1987, Perception & psychophysics.

[47]  Michael S. Landy,et al.  The kinetic depth effect and optic flowȁII. First- and second-order motion , 1991, Vision Research.

[48]  Hiroshi Ando,et al.  Dynamic reconstruction and integration of 3D structure information , 1993 .

[49]  G. E. Mueser,et al.  Accuracy in reconstructing the arrangement of elements generating kinetic depth displays. , 1960, Journal of experimental psychology.

[50]  J. Heel.  Direct Estimation of Structure and Motion from Multiple Frames , 1990 .

[51]  W. Eric L. Grimson,et al.  Surface consistency constraints in vision , 1982, Comput. Graph. Image Process..

[52]  A Blake,et al.  3D model recognition from stereoscopic cues , 1991 .

[53]  M L BRAUNSTEIN,et al.  The perception of depth through motion. , 1962, Psychological bulletin.

[54]  Katsushi Ikeuchi,et al.  Numerical Shape from Shading and Occluding Boundaries , 1981, Artif. Intell..

[55]  Donald D. Hoffman,et al.  Discriminating rigid from nonrigid motion: Minimum points and views , 1990, Perception & psychophysics.

[56]  Claes von Hofsten,et al.  Visual perception of motion in depth: Application of a vector model to three-dot motion patterns , 1973 .

[57]  B. Green.  Figure coherence in the kinetic depth effect. , 1961, Journal of experimental psychology.

[58]  K Nakayama,et al.  Stereoscopic Depth: Its Relation to Image Segmentation, Grouping, and the Recognition of Occluded Objects , 1989, Perception.

[59]  E C Hildreth.  Computational studies of the extraction of visual spatial information from binocular and motion cues. , 1988, Canadian journal of physiology and pharmacology.

[60]  H. Wallach,et al.  The kinetic depth effect. , 1953, Journal of experimental psychology.

[61]  Narendra Ahuja,et al.  EXTRACTING SURFACES FROM STEREO IMAGES: AN INTEGRATED APPROACH. , 1987 .

[62]  L. Schumaker.  Fitting surfaces to scattered data , 1976 .

[63]  G. Johansson.  Visual perception of biological motion and a model for its analysis , 1973 .

[64]  J S Lappin,et al.  Detection of three-dimensional structure in moving optical patterns. , 1984, Journal of experimental psychology. Human perception and performance.

[65]  Shahriar Negahdaripour,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence , 2004 .

[66]  J T Todd,et al.  The perception of 3-dimensional affine structure from minimal apparent motion sequences , 1990, Perception & psychophysics.

[67]  C. Chui,et al.  Approximation Theory II , 1976 .

[68]  Tomaso Poggio,et al.  Probabilistic Solution of Ill-Posed Problems in Computational Vision , 1987 .

[69]  Hugh F. Durrant-Whyte,et al.  Consistent Integration and Propagation of Disparate Sensor Observations , 1986, Proceedings. 1986 IEEE International Conference on Robotics and Automation.

[70]  J H Rieger,et al.  Processing differential image motion. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[71]  R A Andersen,et al.  The response of area MT and V1 neurons to transparent motion , 1991, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[72]  Donald D. Hoffman,et al.  Minimum points and views for the recovery of three-dimensional structure. , 1987, Journal of experimental psychology. Human perception and performance.

[73]  H. Wallach,et al.  Circles and derived figures in rotation. , 1956, The American journal of psychology.

[74]  James E. Cutting,et al.  Blowing in the wind: Perceiving structure in trees and bushes , 1982, Cognition.

[75]  S. Cobb,et al.  Perception of 3-D structure from motion: The role of velocity gradients and segmentation boundaries , 1988, Perception & psychophysics.

[76]  Azriel Rosenfeld,et al.  Human and Machine Vision , 1983 .

[77]  Norberto M. Grzywacz,et al.  A computational theory for the perception of coherent visual motion , 1988, Nature.

[78]  W. Eric L. Grimson,et al.  An implementation of a computational theory of visual surface interpolation , 1983, Comput. Vis. Graph. Image Process..

[79]  H. F. Durrant-White.  Consistent integration and propagation of disparate sensor observations , 1987 .

[80]  W E Grimson,et al.  A computational theory of visual surface interpolation. , 1982, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[81]  Eric L. W. Grimson,et al.  From Images to Surfaces: A Computational Study of the Human Early Visual System , 1981 .

[82]  Berthold K. P. Horn,et al.  Passive navigation , 1982, Comput. Vis. Graph. Image Process..

[83]  David W. Eby,et al.  Perceiving Structure From Motion: Failure Of Shape Constancy , 1988, [1988 Proceedings] Second International Conference on Computer Vision.

[84]  Demetri Terzopoulos,et al.  The Computation of Visible-Surface Representations , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[85]  Joachim Heel.  Dynamic motion vision , 1990, Robotics Auton. Syst..

[86]  Baba C. Vemuri,et al.  On Three-Dimensional Surface Reconstruction Methods , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[87]  Lucia M. Vaina,et al.  Structure from Motion with Impaired Local-Speed and Global Motion-Field Computations , 1990, Neural Computation.

[88]  G J Andersen.  Perception of three-dimensional structure from optic flow without locally smooth velocity. , 1989, Journal of experimental psychology. Human perception and performance.

[89]  J T Todd,et al.  Apparent rotation in three-dimensional space: Effects of temporal, spatial, and structural factors , 1988, Perception & psychophysics.

[90]  Olivier D. Faugeras,et al.  Representing stereo data with the Delaunay triangulation , 1988, Proceedings. 1988 IEEE International Conference on Robotics and Automation.

[91]  Stefan Treue,et al.  Structure-from-motion: Perceptual evidence for surface interpolation , 1995, Vision Research.

[92]  S. P. Mudur,et al.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[93]  W F Clocksin,et al.  Perception of Surface Slant and Edge Labels from Optical Flow: A Computational Approach , 1980, Perception.

[94]  Green Bf.  Figure coherence in the kinetic depth effect. , 1961 .

[95]  Valdis Berzins,et al.  Dynamic Occlusion Analysis in Optical Flow Fields , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[96]  Myron L. Braunstein,et al.  Depth perception through motion , 1976 .

[97]  Stefan Treue,et al.  Surface Interpolation in Three-Dimensional Structure-from-Motion Perception , 1989, Neural Computation.

[98]  Ellen C. Hildreth,et al.  The perceptual buildup of three-dimensional structure from motion , 1989, Perception & psychophysics.

[99]  W. Eric L. Grimson,et al.  Computational Experiments with a Feature Based Stereo Algorithm , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[100]  Hormoz Shariat,et al.  Motion Estimation with More than Two Frames , 1990, IEEE Trans. Pattern Anal. Mach. Intell..