Through-the-Lens Multi-camera Synchronisation and Frame-Drop Detection for 3D Reconstruction

Synchronisation is an essential requirement for multiview 3D reconstruction of dynamic scenes. However, the use of HD cameras and large set-ups put a considerable stress on hardware and cause frame drops, which is usually detected by manually verifying very large amounts of data. This paper improves [9], and extends it with frame-drop detection capability. In order to spot frame-drop events, the algorithm fits a broken line to the frame index correspondences for each camera pair, and then fuses the pair wise drop hypotheses into a consistent, absolute frame-drop estimate. The success and the practical utility of the the improved pipeline is demonstrated through a number of experiments, including 3D reconstruction and free-viewpoint video rendering tasks.

[1]  Yael Moses,et al.  Video Synchronization Using Temporal Signals from Epipolar Lines , 2010, ECCV.

[2]  Hans-Peter Seidel,et al.  Markerless Motion Capture with unsynchronized moving cameras , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Cheng Lei,et al.  Tri-focal tensor-based multiple video synchronization with subframe optimization , 2006, IEEE Transactions on Image Processing.

[4]  Marc Pollefeys,et al.  Camera Network Calibration and Synchronization from Silhouettes in Archived Video , 2010, International Journal of Computer Vision.

[5]  PollefeysMarc,et al.  Camera Network Calibration and Synchronization from Silhouettes in Archived Video , 2010 .

[6]  Adrian Hilton,et al.  Through-the-Lens Synchronisation for Heterogeneous Camera Networks , 2012, BMVC.

[7]  Cordelia Schmid,et al.  A performance evaluation of local descriptors , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Xin Li,et al.  Subframe Video Synchronization via 3D Phase Correlation , 2006, 2006 International Conference on Image Processing.

[9]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[10]  Marcus A. Magnor,et al.  Subframe Temporal Alignment of Non-Stationary Cameras , 2008, BMVC.

[11]  Adrian Hilton,et al.  Wand-based Multiple Camera Studio Calibration , 2007 .

[12]  Luc Van Gool,et al.  Synchronizing video sequences , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[13]  Andrew W. Fitzgibbon,et al.  The Problem of Degeneracy in Structure and Motion Recovery from Uncalibrated Image Sequences , 1999, International Journal of Computer Vision.

[14]  Ian D. Reid,et al.  Video synchronization from human motion using rank constraints , 2009, Comput. Vis. Image Underst..

[15]  M. Pollefeys,et al.  VIDEO SYNCHRONIZATION VIA SPACE-TIME INTEREST POINT DISTRIBUTION , 2004 .

[16]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[17]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[18]  Jiri Matas,et al.  Optimal Randomized RANSAC , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Tanveer F. Syeda-Mahmood,et al.  View-invariant alignment and matching of video sequences , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[20]  Lior Wolf,et al.  Wide Baseline Matching between Unsynchronized Video Sequences , 2006, International Journal of Computer Vision.

[21]  Joan Serrat,et al.  Video Alignment for Change Detection , 2011, IEEE Transactions on Image Processing.

[22]  Jean-Yves Guillemaut,et al.  Objective Quality Assessment in Free-Viewpoint Video Production , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[23]  Denis Simakov,et al.  Feature-Based Sequence-to-Sequence Matching , 2006, International Journal of Computer Vision.

[24]  Jean-Yves Guillemaut,et al.  Calibration of Nodal and Free-Moving Cameras in Dynamic Scenes for Post-Production , 2011, 2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission.

[25]  Kiriakos N. Kutulakos,et al.  Linear Sequence-to-Sequence Alignment , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Jean-Yves Guillemaut,et al.  Joint Multi-Layer Segmentation and Reconstruction for Free-Viewpoint Video Applications , 2011, International Journal of Computer Vision.