Camera–projector matching using unstructured video

This paper presents a novel approach for matching 2-D points between a video projector and a digital camera. Our method is motivated by camera–projector applications for which the projected image needs to be warped to prevent geometric distortion. Since the warping process often needs geometric information on the 3-D scene obtained from a triangulation, we propose a technique for matching points in the projector to points in the camera based on arbitrary video sequences. The novelty of our method lies in the fact that it does not require the use of pre-designed structured light patterns as is usually the case. The backbone of our application lies in a function that matches activity patterns instead of colors. This makes our method robust to pose, severe photometric and geometric distortions. It also does not require calibration of the color response curve of the camera–projector system. We present quantitative and qualitative results with synthetic and real-life examples, and compare the proposed method with the scale invariant feature transform (SIFT) method and with a state-of-the-art structured light technique. We show that our method performs almost as well as structured light methods and significantly outperforms SIFT when the contrast of the video captured by the camera is degraded.

[1]  Vaughan R. Pratt,et al.  Direct least-squares fitting of algebraic surfaces , 1987, SIGGRAPH.

[2]  Takeo Kanade,et al.  Stereo by Intra- and Inter-Scanline Search Using Dynamic Programming , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  R. Hartley Triangulation, Computer Vision and Image Understanding , 1997 .

[4]  Eric Li,et al.  Dynamic projection environments for immersive visualization , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[5]  H. Hirschmüller Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information , 2005, CVPR.

[6]  Tania Landes,et al.  AUTOMATIC SEGMENTATION OF BUILDING FACADES USING TERRESTRIAL LASER DATA , 2007 .

[7]  Greg Welch,et al.  A Distributed Cooperative Framework for Continuous Multi-Projector Pose Estimation , 2009, 2009 IEEE Virtual Reality Conference.

[8]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  A. Verri,et al.  A compact algorithm for rectification of stereo pairs , 2000 .

[10]  Li Zhang,et al.  Rapid shape acquisition using color structured light and multi-pass dynamic programming , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[11]  Markus H. Gross,et al.  Adaptive Instant Displays: Continuously Calibrated Projections Using Per‐Pixel Light Control , 2005, Comput. Graph. Forum.

[12]  Geoffrey Egnal,et al.  Detecting Binocular Half-Occlusions: Empirical Comparisons of Five Approaches , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Henry Fuchs,et al.  Real-Time Projector Tracking on Complex Geometry Using Ordinary Imagery , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Robert E. Mahony,et al.  Spatio-Temporal RANSAC for Robust Estimation of Ground Plane in Video Range Images for Automotive Applications , 2008, 2008 11th International IEEE Conference on Intelligent Transportation Systems.

[15]  Oliver Bimber,et al.  Passive-Active Geometric Calibration for View-Dependent Projections onto Arbitrary Surfaces , 2007, J. Virtual Real. Broadcast..

[16]  Pierre-Marc Jodoin,et al.  Camera-projector matching using an unstructured video stream , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[17]  Ruigang Yang,et al.  Multi-projector display with continuous self-calibration , 2008, PROCAMS '08.

[18]  Joaquim Salvi,et al.  Pattern codification strategies in structured light systems , 2004, Pattern Recognit..

[19]  N. Otsu A threshold selection method from gray level histograms , 1979 .

[20]  B. Wüthrich AN INTERNATIONAL SYMPOSIUM , 1997 .

[21]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[22]  Li Zhang,et al.  Spacetime stereo: shape recovery for dynamic scenes , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[23]  G. A. Watson,et al.  Algebraic Fitting of Quadric Surfaces to Data , 2005 .

[24]  Mark Ashdown,et al.  Steerable Projector Calibration , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[25]  Shree K. Nayar,et al.  Projection defocus analysis for scene capture and image display , 2006, SIGGRAPH 2006.

[26]  Shree K. Nayar,et al.  Rectifying transformations that minimize resampling effects , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[27]  Henry Fuchs,et al.  RANSAC-Assisted Display Model Reconstruction for Projective Display , 2006, IEEE Virtual Reality Conference (VR 2006).

[28]  Szymon Rusinkiewicz,et al.  Spacetime Stereo: A Unifying Framework for Depth from Triangulation , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Alexandru Tupan,et al.  Triangulation , 1997, Comput. Vis. Image Underst..

[30]  Pedro F. Felzenszwalb,et al.  Efficient belief propagation for early vision , 2004, CVPR 2004.

[31]  Paul A. Beardsley,et al.  A self-correcting projector , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[32]  Peter N. Belhumeur,et al.  A Bayesian approach to binocular steropsis , 1996, International Journal of Computer Vision.

[33]  Paul S. Heckbert Color image quantization for frame buffer display , 1998 .

[34]  Mark Fiala,et al.  ARTag, a fiducial marker system using digital techniques , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[35]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[36]  Sébastien Roy,et al.  Multi-projectors for arbitrary surfaces without explicit calibration nor reconstruction , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[37]  Sébastien Roy,et al.  Fast multiple-baseline stereo with occlusion , 2005, Fifth International Conference on 3-D Digital Imaging and Modeling (3DIM'05).

[38]  Ruigang Yang Automatic Projector Display Surface Estimation Using Every-Day Imagery , 2010 .

[39]  Matthijs C. Dorst Distinctive Image Features from Scale-Invariant Keypoints , 2011 .