Panoramic video stitching from commodity HDTV cameras

Digital camera and smartphone technologies have made high-quality images and video pervasive and abundant. Combining or stitching collections of images from a variety of viewpoints into an extended panoramic image is a common and popular function for such devices. Extending this functionality to video however, poses many new challenges due to the demand for both spatial and temporal continuity. Multi-view video stitching (also called panoramic video stitching) is an emerging, common research area in computer vision, image/video processing and computer graphics and has wide applications in virtual reality, virtual tourism, surveillance, and human computer interaction. In this paper, we study and solve the major technical and practical problems in the complete process of stitching a high-resolution multi-view video into a high-resolution panoramic video. The challenges addressed include video stabilization, efficient high-definition multi-view video alignment and stitching, color correction, and blurred frame detection and repair. The proposed approaches have been successfully applied in a high-quality virtual reality system—the Virtual Exercise Environment (VEE) system.

[1]  P. Jansson.  Deconvolution of images and spectra , 1997 .

[2]  Zhongliang Jing,et al.  Aerial sequence image mosaic using reduced sift descriptors , 2007, International Symposium on Multispectral Image Processing and Pattern Recognition.

[3]  Ayman Kaheel,et al.  Stitching videos streamed by mobile phones in real-time , 2009, ACM Multimedia.

[4]  K. C. A. Smith,et al.  An automatic focusing and astigmatism correction system for the SEM and CTEM , 1982 .

[5]  Wei Xu,et al.  Augmenting Exercise Systems with Virtual Exercise Environment , 2009, ISVC.

[6]  Zhonghua Liu,et al.  Face Recognition Based on Wavelet Transform, Singular Value Decomposition and Kernel Principal Component Analysis , 2008, 2008 Chinese Conference on Pattern Recognition.

[7]  M. Irani,et al.  Spatio-Temporal Alignment of Sequences , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Y. Fisher.  Fractal image compression: theory and application , 1995 .

[9]  Fuyuan Peng,et al.  Fast image matching for localization in deep-sea based on the simplified SIFT (scale invariant feature transform) algorithm , 2007, Other Conferences.

[10]  A. Ardeshir Goshtasby,et al.  2-D and 3-D Image Registration , 2004 .

[11]  P. Jansson.  Deconvolution of images and spectra (2nd ed.) , 1996 .

[12]  Richard Szeliski,et al.  Image Restoration by Matching Gradient Distributions , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Ankit Gupta,et al.  Single Image Deblurring Using Motion Density Functions , 2010, ECCV.

[14]  AnguelovDragomir,et al.  Google Street View , 2010 .

[15]  Jason P. de Villiers,et al.  Real-time photogrammetric stitching of high resolution video on COTS hardware , 2009 .

[16]  Richard Szeliski,et al.  Video mosaics for virtual environments , 1996, IEEE Computer Graphics and Applications.

[17]  Shangbo Zhou,et al.  A Fast SIFT Feature Matching Algorithm for Image Registration , 2011, 2011 International Conference on Multimedia and Signal Processing.

[18]  Stephen Lin,et al.  Single-image vignetting correction using radial gradient symmetry , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Christian Früh,et al.  Google Street View: Capturing the World at Street Level , 2010, Computer.

[20]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[21]  Matthew A. Brown,et al.  Automatic Panoramic Image Stitching using Invariant Features , 2007, International Journal of Computer Vision.

[22]  Richard Szeliski,et al.  Image mosaicing for tele-reality applications , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[23]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[24]  K Cook,et al.  Comparison of autofocus methods for automated microscopy. , 1991, Cytometry.

[25]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[26]  D. Simon.  Optimal State Estimation: Kalman, H Infinity, and Nonlinear Approaches , 2006 .

[27]  Ramesh Raskar,et al.  Coded exposure photography: motion deblurring using fluttered shutter , 2006, SIGGRAPH 2006.

[28]  Harry Shum,et al.  Full-frame video stabilization with motion inpainting , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Ayman Kaheel,et al.  Mobicast: a system for collaborative event casting using mobile phones , 2009, MUM.

[30]  Shmuel Peleg,et al.  Seamless Image Stitching in the Gradient Domain , 2004, ECCV.

[31]  Weisi Lin,et al.  A no-reference quality metric for measuring image blur , 2003, Seventh International Symposium on Signal Processing and Its Applications, 2003. Proceedings..

[32]  Jyoti Singhai,et al.  Review of Motion Estimation and Video Stabilization techniques For hand held mobile video , 2011 .

[33]  Ayman Kaheel,et al.  Fast stitching of videos captured from freely moving devices by exploiting temporal redundancy , 2010, 2010 IEEE International Conference on Image Processing.

[34]  Matthew A. Brown,et al.  Recognising panoramas , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[35]  Wei Xu,et al.  Detecting and classifying blurred image regions , 2013, 2013 IEEE International Conference on Multimedia and Expo (ICME).

[36]  Li Xu,et al.  Two-Phase Kernel Estimation for Robust Motion Deblurring , 2010, ECCV.

[37]  Wei Xu,et al.  Performance evaluation of color correction approaches for automatic multi-view image and video stitching , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[38]  Marcelo H. Ang,et al.  Practical issues in pixel-based autofocusing for machine vision , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[39]  Axel Gräser,et al.  VF-SIFT: Very Fast SIFT Feature Matching , 2010, DAGM-Symposium.

[40]  Yuchi Xu,et al.  A new approach to video stabilization with iterative smoothing , 2010, IEEE 10th INTERNATIONAL CONFERENCE ON SIGNAL PROCESSING PROCEEDINGS.

[41]  Peter Reinartz,et al.  Combining Mutual Information and Scale Invariant Feature Transform for Fast and Robust Multisensor SAR Image Registration , 2009 .

[42]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[43]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[44]  Sunglok Choi,et al.  Robust video stabilization to outlier motion using adaptive RANSAC , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[45]  Janusz Konrad,et al.  Probabilistic video stabilization using Kalman filtering and mosaicing , 2003, IS&T/SPIE Electronic Imaging.

[46]  Yan Ke,et al.  PCA-SIFT: a more distinctive representation for local image descriptors , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[47]  Jiaya Jia,et al.  High-quality motion deblurring from a single image , 2008, ACM Trans. Graph..

[48]  Yiding Wang,et al.  SIFT Based Automatic Tie-Point Extraction for Multitemporal SAR Images , 2008, 2008 International Workshop on Education Technology and Training & 2008 International Workshop on Geoscience and Remote Sensing.

[49]  Tien Tsin,et al.  Image Partial Blur Detection and Classification , 2013 .

[50]  Anat Levin,et al.  Blind Motion Deblurring Using Image Statistics , 2006, NIPS.

[51]  Maurizio Pilu,et al.  Video stabilization as a variational problem and numerical solution with the Viterbi method , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[52]  Ayman Kaheel,et al.  Improved optimal seam selection blending for fast video stitching of videos captured from freely moving devices , 2011, 2011 18th IEEE International Conference on Image Processing.