Geometry-aware Video Registration

We present a new method for the accurate registration of video sequences of a real object over its dense triangular mesh. The goal is to obtain an accurate video-to-geometry registration to allow the bidirectional data transfer between the 3D model and the video using the perspective projection defined by the camera model. Our solution uses two different approaches: feature-based registration by KLT video tracking, and statistic-based registration by maximizing the Mutual Information (MI) between the gradient of the frame and the gradient of the rendering of the 3D model with some illumination related properties, such as surface normals and ambient occlusion. While the first approach allows a fast registration of short sequences with simple camera movements, the MI is used to correct the drift problem that KLT tracker produces over long sequences, due to the incremental tracking and the camera motion. We demonstrate, using synthetic sequences, that the alignment error obtained with our method is smaller than the one introduced by KLT, and we show the results of some interesting and challenging real sequences of objects of different sizes, acquired under different conditions.

[1]  Hans-Peter Seidel,et al.  Automated texture registration and stitching for real world models , 2000, Proceedings the Eighth Pacific Conference on Computer Graphics and Applications.

[2]  Guopu Zhu,et al.  Efficient Illumination Insensitive Object Tracking by Normalized Gradient Matching , 2007, IEEE Signal Processing Letters.

[3]  M. Powell.  Developments of NEWUOA for minimization without derivatives , 2008 .

[4]  Konrad Klein,et al.  Texturing 3D Models of Real World Objects from Multiple Unregistered Photographic Views , 1999, Comput. Graph. Forum.

[5]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[6]  Paolo Cignoni,et al.  Minimizing user intervention in registering 2D images to 3D models , 2005, The Visual Computer.

[7]  George Wolberg,et al.  Multiview Geometry for Texture Mapping 2D Images Onto 3D Range Data , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[8]  Stefano Soatto,et al.  Real-time feature tracking and outlier rejection with changes in illumination , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[9]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[10]  Luc Van Gool,et al.  Wide Baseline Stereo Matching based on Local, Affinely Invariant Regions , 2000, BMVC.

[11]  Luc Van Gool,et al.  Drift detection and removal for sequential structure from motion algorithms , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Paul A. Viola,et al.  Alignment by Maximization of Mutual Information , 1997, International Journal of Computer Vision.

[13]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Max A. Viergever,et al.  Mutual-information-based registration of medical images: a survey , 2003, IEEE Transactions on Medical Imaging.

[15]  Natasha Gelfand,et al.  SURFTrac: Efficient tracking and continuous object recognition using local feature descriptors , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Roberto Scopigno,et al.  Image‐to‐Geometry Registration: a Mutual Information Method exploiting Illumination‐related Geometric Properties , 2009, Comput. Graph. Forum.

[17]  Cordelia Schmid,et al.  Indexing Based on Scale Invariant Interest Points , 2001, ICCV.

[18]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[19]  Guy Marchal,et al.  Multimodality image registration by maximization of mutual information , 1997, IEEE Transactions on Medical Imaging.

[20]  Eric Foxlin,et al.  Encoded LED system for optical trackers , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[21]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[22]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[23]  Kok-Lim Low,et al.  Automatic registration of color images to 3D geometry , 2009, CGI.

[24]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[25]  Reinhard Koch,et al.  Visual Modeling with a Hand-Held Camera , 2004, International Journal of Computer Vision.

[26]  Hans-Peter Seidel,et al.  Merging of Feature Tracks for Camera Motion Estimation from Video , 2008 .

[27]  Éric Marchand,et al.  Optimal detection and tracking of feature points using mutual information , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).