Depth and Scene Flow from a Single Moving Camera

We show that it is possible to reconstruct scene flow and depth from a single moving camera whose motion is known. To do so, we assume that the local scene motion can be approximated by a constant velocity in a small temporal window. This assumption makes it possible to unambiguously reconstruct scene flow and depth using as few as 3 frames from the sequence. We propose a variational approach to directly solve for structure and flow, and we demonstrate the results on challenging real-world data with both rigid and non-rigid motion. The experiments illustrate that the inclusion of flow in the case of non-rigid motion allows us to reconstruct a better geometry than if motion was simply ignored.

[1]  S Ullman,et al.  Maximizing Rigidity: The Incremental Recovery of 3-D Structure from Rigid and Nonrigid Motion , 1984, Perception.

[2]  Aaron Hertzmann,et al.  Learning Non-Rigid 3D Shape from 2D Motion , 2003, NIPS.

[3]  Amnon Shashua,et al.  Trajectory Triangulation: 3D Reconstruction of Moving Points from a Monocular Image Sequence , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Frederic Devernay,et al.  A Variational Method for Scene Flow Estimation from Stereo Sequences , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[5]  Hujun Bao,et al.  Recovering consistent video depth maps via bundle optimization , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Ye Zhang,et al.  Integrated 3D scene flow and structure recovery from multiview image sequences , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[7]  Takeo Kanade,et al.  Image-based spatio-temporal modeling and view interpolation of dynamic events , 2005, TOGS.

[8]  Kiriakos N. Kutulakos,et al.  Multi-View Scene Capture by Surfel Sampling: From Video Streams to Non-Rigid 3D Motion, Shape and Reflectance , 2002, International Journal of Computer Vision.

[9]  Olivier D. Faugeras,et al.  Modelling dynamic scenes by registering multi-view image sequences , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[10]  Daniel Cremers,et al.  Efficient Dense Scene Flow from Sparse or Dense Stereo Data , 2008, ECCV.

[11]  Frederic Devernay,et al.  Multi-Camera Scene Flow by Tracking 3-D Points and Surfels , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[12]  Cedric Nishan Canagarajah,et al.  Time Varying Volumetric Scene Reconstruction Using Scene Flow , 2007, BMVC.

[13]  José Miguel Buenaposada,et al.  Real-time tracking and estimation of plane pose , 2002, Object recognition supported by user interaction for service robots.

[14]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[15]  Takeo Kanade,et al.  Three-dimensional scene flow , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Henning Biermann,et al.  Recovering non-rigid 3D shape from image streams , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[17]  Takeo Kanade,et al.  Shape and motion carving in 6D , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[18]  Vincent Lepetit,et al.  Closed-Form Solution to Non-rigid 3D Surface Registration , 2008, ECCV.

[19]  Li Zhang,et al.  Spacetime stereo: shape recovery for dynamic scenes , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[20]  Rui Li,et al.  Multi-Scale 3D Scene Flow from Binocular Stereo Sequences , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[21]  Joachim Weickert,et al.  Towards ultimate motion estimation: combining highest accuracy with real-time performance , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.