Panoramic video stitching

Digital camera and smartphone technologies have made high quality images and video pervasive and abundant. Combining or stitching collections of images from a variety of viewpoints into an extended panoramic image is a common and popular function for such devices. Extending this functionality to video however, poses many new challenges due to the demand for both spatial and temporal continuity. Multi-view video stitching (also called panoramic video stitching) is an emerging, common research area in computer vision, image/video processing and computer graphics and has wide applications in virtual reality, virtual tourism, surveillance, and human computer interaction. In this thesis, I will explore the technical and practical problems in the complete process of stitching a high-resolution multi-view video into a high-resolution panoramic video. The challenges addressed include video stabilization, efficient multi-view video alignment and panoramic video stitching, color correction, and blurred frame detection and repair. Specifically, I propose a continuity aware Kalman filtering scheme for rotation angles for video stabilization and jitter removal. For efficient stitching of long, high-resolution panoramic videos, I propose constrained and multi-grid SIFT matching schemes, concatenated image projection and warping and min-space feathering. These three approaches together can greatly reduce the computational time and memory requirement in panoramic video stitching, which makes it feasible to stitch high-resolution (e.g., 1920x1080 pixels) and long panoramic video sequences using standard workstations. Color correction is the emphasis of my research. On this topic I first performed a systematic survey and performance evaluation of nine state of the art color correction approaches in the context of two-view image stitching. My evaluation work not only gives useful insights and conclusions about the relative performance of these approaches, but also points out the remaining challenges and possible directions for future color correction research. Based on the conclusions from this evaluation work, I proposed a hybrid and scalable color correction approach for general n-view image stitching, and designed a two-view video color correction approach for panoramic video stitching. For blurred frame detection and repair, I have completed preliminary work on image partial blur detection and classification, in which I proposed a SVM-based blur block classifier using improved and new local blur features. Then, based on partial blur classification results, I designed a statistical thresholding scheme for blurred frame identification. For the detected blurred frames, I repaired them using polynomial data fitting from neighboring unblurred frames. Many of the techniques and ideas in this thesis are novel and general solutions to the technical or practical problems in panoramic video stitching. At the end of this thesis, I conclude the contributions made by this thesis to the research and popularization of panoramic video stitching, and describe those open research issues.

[1]  Christian Früh,et al.  Google Street View: Capturing the World at Street Level , 2010, Computer.

[2]  Franco Oberti,et al.  A new sharpness metric based on local kurtosis, edge and energy information , 2004, Signal Process. Image Commun..

[3]  Harry Shum,et al.  Full-frame video stabilization with motion inpainting , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Lizhuang Ma,et al.  Color transfer in correlated color space , 2006, VRCIA '06.

[5]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[6]  Kenji Yamamoto,et al.  Color correction for multi-view video using energy minimization of view networks , 2008, Int. J. Autom. Comput..

[7]  Cordelia Schmid,et al.  A Performance Evaluation of Local Descriptors , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Eli Shechtman,et al.  Space-time video completion , 2004, CVPR 2004.

[9]  Yingen Xiong,et al.  Mobile panoramic imaging system , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[10]  Harpreet S. Sawhney,et al.  True Multi-Image Alignment and Its Application to Mosaicing and Lens Distortion Correction , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Tai-Pang Wu,et al.  Video repairing: inference of foreground and background under severe occlusion , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[12]  A.C. Kokaram,et al.  N-dimensional probability density function transfer and its application to color transfer , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[13]  Ross Cutler,et al.  Practical calibrations for a real-time digital omnidirectional camera , 2001 .

[14]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[15]  K. Pulli,et al.  Color Matching of Image Sequences with Combined Gamma and Linear Corrections , 2010 .

[16]  Weisi Lin,et al.  A no-reference quality metric for measuring image blur , 2003, Seventh International Symposium on Signal Processing and Its Applications, 2003. Proceedings..

[17]  Greg Welch,et al.  Ensuring color consistency across multiple cameras , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[18]  Yingen Xiong,et al.  Color matching for high-quality panoramic images on mobile phones , 2010, IEEE Transactions on Consumer Electronics.

[19]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[20]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[21]  Guozhi Tao,et al.  Symmetric inverse consistent nonlinear registration driven by mutual information , 2009, Comput. Methods Programs Biomed..

[22]  Yuan Cheng,et al.  Correcting over-exposure in photographs , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[23]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[24]  Michael J. Black,et al.  Fields of Experts: a framework for learning image priors , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[25]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[26]  Wei Xu,et al.  2D Barcode localization and motion deblurring using a flutter shutter camera , 2011, 2011 IEEE Workshop on Applications of Computer Vision (WACV).

[27]  Brian Bouzas,et al.  Objective image quality measure derived from digital image power spectra , 1992 .

[28]  Steve Mann,et al.  Virtual bellows: constructing high quality stills from video , 1994, Proceedings of 1st International Conference on Image Processing.

[29]  Sunglok Choi,et al.  Robust video stabilization to outlier motion using adaptive RANSAC , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[30]  D J Field,et al.  Relations between the statistics of natural images and the response properties of cortical cells. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[31]  Richard Szeliski,et al.  Eliminating ghosting and exposure artifacts in image mosaics , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[32]  Wei Xu,et al.  Feature Matching under Region-Based Constraints for Robust Epipolar Geometry Estimation , 2009, ISVC.

[33]  Cordelia Schmid,et al.  Scale & Affine Invariant Interest Point Detectors , 2004, International Journal of Computer Vision.

[34]  Zhigang Zhu.  CAMERA STABILIZATION BASED ON 2.5D MOTION ESTIMATION AND INERTIAL MOTION FILTERING , 1998 .

[35]  James Davis,et al.  Mosaics of scenes with moving objects , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[36]  Wei-Ying Ma,et al.  Blur determination in the compressed domain using DCT information , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[37]  Mariano Rivera,et al.  Computing the α-channel with probabilistic segmentation for image colorization , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[38]  Matthew A. Brown,et al.  Recognising panoramas , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[39]  Hubert Konik,et al.  Automatic blur detection for meta-data extraction in content-based retrieval context , 2003, IS&T/SPIE Electronic Imaging.

[40]  Xiaowei Han,et al.  A Mosaic Approach for Remote Sensing Images Based on Wavelet Transform , 2008, 2008 4th International Conference on Wireless Communications, Networking and Mobile Computing.

[41]  Chi-Keung Tang,et al.  Eliminating structure and intensity misalignment in image stitching , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[42]  Maurizio Pilu,et al.  Video stabilization as a variational problem and numerical solution with the Viterbi method , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[43]  Sabine Süsstrunk,et al.  Mapping colour in image stitching applications , 2004, J. Vis. Commun. Image Represent..

[44]  Frank M. Candocia.  Simultaneous homographic and comparametric alignment of multiple exposure-adjusted pictures of the same scene , 2003, IEEE Trans. Image Process..

[45]  Y. Fisher.  Fractal image compression: theory and application , 1995 .

[46]  Jean Ponce,et al.  Computer Vision: A Modern Approach , 2002 .

[47]  P. Anandan,et al.  Hierarchical Model-Based Motion Estimation , 1992, ECCV.

[48]  Shenchang Eric Chen,et al.  QuickTime VR: an image-based approach to virtual environment navigation , 1995, SIGGRAPH.

[49]  Hsien-Che Lee,et al.  Introduction to Color Imaging Science: Introduction , 2005 .

[50]  Michael Bosse,et al.  Non-metric image-based rendering for video stabilization , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[51]  Yoav Y. Schechner,et al.  Addressing radiometric nonidealities: a unified framework , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[52]  Don Kimber,et al.  Enhancing distance learning with panoramic video , 2001, Proceedings of the 34th Annual Hawaii International Conference on System Sciences.

[53]  B. Funt,et al.  Diagonal versus affine transformations for color correction. , 2000, Journal of the Optical Society of America. A, Optics, image science, and vision.

[54]  Andrew Zisserman,et al.  Automated mosaicing with super-resolution zoom , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[55]  Jiri Matas,et al.  Robust wide-baseline stereo from maximally stable extremal regions , 2004, Image Vis. Comput..

[56]  Sei-Wang Chen,et al.  Video stabilization for a hand-held camera based on 3D motion model , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[57]  Wei Chen,et al.  A Novel Variational Image Model: Towards a Unified Approach to Image Editing , 2006, Journal of Computer Science and Technology.

[58]  Qiang Wu,et al.  Motion blur detection based on lowest directional high-frequency energy , 2010, 2010 IEEE International Conference on Image Processing.

[59]  Marc Pollefeys,et al.  Robust Radiometric Calibration and Vignetting Correction , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  Michael T. Postek,et al.  A Kurtosis-Based Statistical Measure for Two-Dimensional Processes and Its Applications to Image Sharpness , 2003 .

[61]  Richard Szeliski,et al.  Video mosaics for virtual environments , 1996, IEEE Computer Graphics and Applications.

[62]  Richard Szeliski,et al.  Multi-image matching using multi-scale oriented patches , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[63]  Ingeborg Tastl,et al.  Sharpness measure: towards automatic image enhancement , 2005, IEEE International Conference on Image Processing 2005.

[64]  Ying Wu,et al.  Estimating space-variant motion blur without deblurring , 2008, 2008 15th IEEE International Conference on Image Processing.

[65]  Janusz Konrad,et al.  Probabilistic video stabilization using Kalman filtering and mosaicing , 2003, IS&T/SPIE Electronic Imaging.

[66]  Lina J. Karam,et al.  No-reference objective wavelet based noise immune image sharpness metric , 2005, IEEE International Conference on Image Processing 2005.

[67]  Yuchi Xu,et al.  A new approach to video stabilization with iterative smoothing , 2010, IEEE 10th INTERNATIONAL CONFERENCE ON SIGNAL PROCESSING PROCEEDINGS.

[68]  Dan B. Goldman,et al.  Vignette and exposure calibration and compensation , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[69]  Miguel Oliveira,et al.  Unsupervised local color correction for coarsely registered images , 2011, CVPR 2011.

[70]  Shmuel Peleg,et al.  Seamless Image Stitching in the Gradient Domain , 2004, ECCV.

[71]  Hanghang Tong,et al.  Blur detection for digital images using wavelet transform , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[72]  Adrien Bartoli,et al.  A Framework for Pencil-of-Points Structure-from-Motion , 2004, ECCV.

[73]  Gunther Wyszecki,et al.  Color Science: Concepts and Methods, Quantitative Data and Formulae, 2nd Edition , 2000 .

[74]  Rachid Deriche,et al.  Using geometric corners to build a 2D mosaic from a set of images , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[75]  Hong Li,et al.  Selective color transfer with multi-source images , 2009, Pattern Recognit. Lett..

[76]  Richard Szeliski,et al.  Systems and Experiment Paper: Construction of Panoramic Image Mosaics with Global and Local Alignment , 2000, International Journal of Computer Vision.

[77]  Stefan Winkler,et al.  Perceptual blur and ringing metrics: application to JPEG2000 , 2004, Signal Process. Image Commun..

[78]  E. M. Lowry,et al.  Sine-Wave Response of the Visual System. II. Sine-Wave and Square-Wave Contrast Sensitivity*† , 1962 .

[79]  Matthew A. Brown,et al.  Multi-image matching using invariant features , 2005 .

[80]  Lina J. Karam,et al.  A No-Reference Objective Image Sharpness Metric Based on the Notion of Just Noticeable Blur (JNB) , 2009, IEEE Transactions on Image Processing.

[81]  Gui Yun Tian,et al.  Colour correction for panoramic imaging , 2002, Proceedings Sixth International Conference on Information Visualisation.

[82]  Wei Xu,et al.  Performance evaluation of color correction approaches for automatic multi-view image and video stitching , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[83]  Christopher Batten.  Autofocusing and Astigmatism Correction in the Scanning Electron Microscope , 2000 .

[84]  Michael F. Cohen,et al.  GradientShop: A gradient-domain optimization framework for image and video filtering , 2010, TOGS.

[85]  Stephen Lin,et al.  Single-image vignetting correction using radial gradient symmetry , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[86]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[87]  William T. Freeman,et al.  Removing camera shake from a single photograph , 2006, SIGGRAPH 2006.

[88]  G. J. Burton,et al.  Color and spatial structure in natural scenes. , 1987, Applied optics.

[89]  Richard Szeliski,et al.  Creating full view panoramic image mosaics and texture-mapped models , 1997, International Conference on Computer Graphics and Interactive Techniques.

[90]  Matthew A. Brown,et al.  Automatic Panoramic Image Stitching using Invariant Features , 2007, International Journal of Computer Vision.

[91]  Andrew Zisserman,et al.  An Affine Invariant Salient Region Detector , 2004, ECCV.

[92]  Toshiaki Fujii,et al.  Color correction for multi-camera system by using correspondences , 2006, SIGGRAPH '06.

[93]  Li Xu,et al.  Two-Phase Kernel Estimation for Robust Motion Deblurring , 2010, ECCV.

[94]  Philip F. McLauchlan,et al.  Image mosaicing using sequential bundle adjustment , 2002, Image Vis. Comput..

[95]  Stephen Lin,et al.  Single-Image Vignetting Correction , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[96]  Richard Szeliski,et al.  Seamless Image Stitching of Scenes with Large Motions and Exposure Differences , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[97]  Peng Wu,et al.  Detection of Out-Of-Focus Digital Photographs , 2005 .

[98]  Brendan J. Frey,et al.  Video Epitomes , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[99]  Richard Szeliski,et al.  Image mosaicing for tele-reality applications , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[100]  Chi-Keung Tang,et al.  Tensor voting for image correction by global and local intensity alignment , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[101]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[102]  Zygmunt Pizlo,et al.  An investigation of perceived sharpness and sharpness metrics , 2005, IS&T/SPIE Electronic Imaging.

[103]  Azriel Rosenfeld,et al.  Shooting a smooth video with a shaky camera , 2003, Machine Vision and Applications.

[104]  Lei Zhou,et al.  A Linear Color Correction Method for Compressed Images and Videos , 2006, IEICE Trans. Inf. Syst..

[105]  Harry Shum,et al.  Bayesian Correction of Image Intensity with Spatial Consideration , 2004, ECCV.

[106]  Wenbin Chen,et al.  Video Stabilization Using Scale-Invariant Features , 2007, 2007 11th International Conference Information Visualization (IV '07).

[107]  D. Simon.  Optimal State Estimation: Kalman, H Infinity, and Nonlinear Approaches , 2006 .

[108]  Helen C. Shen,et al.  Enforcing stochastic inverse consistency in non-rigid image registration and matching , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[109]  Kristin J. Dana,et al.  Real-time scene stabilization and mosaic construction , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[110]  Ramesh Raskar,et al.  Coded exposure photography: motion deblurring using fluttered shutter , 2006, SIGGRAPH 2006.

[111]  Thomas Serre,et al.  A Theory of Object Recognition: Computations and Circuits in the Feedforward Path of the Ventral Stream in Primate Visual Cortex , 2005 .

[112]  Gregory Dudek,et al.  Rotation and zooming in image mosaicing , 1998, Proceedings Fourth IEEE Workshop on Applications of Computer Vision. WACV'98 (Cat. No.98EX201).

[113]  E. Helmer,et al.  Cloud-Free Satellite Image Mosaics with Regression Trees and Histogram Matching. , 2005 .

[114]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[115]  Luo Juan,et al.  SURF applied in panorama image stitching , 2010, 2010 2nd International Conference on Image Processing Theory, Tools and Applications.

[116]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[117]  Wen Gao,et al.  A no-reference perceptual blur metric using histogram of gradient profile sharpness , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[118]  Wei Xu,et al.  Augmenting Exercise Systems with Virtual Exercise Environment , 2009, ISVC.

[119]  Andrea Fusiello,et al.  Image stabilization by features tracking , 1999, Proceedings 10th International Conference on Image Analysis and Processing.

[120]  Chi-Keung Tang,et al.  Image registration with global and local luminance alignment , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[121]  A. Ardeshir Goshtasby,et al.  2-D and 3-D Image Registration , 2004 .

[122]  Chi-Keung Tang,et al.  Local color transfer via probabilistic segmentation by expectation-maximization , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[123]  K Cook,et al.  Comparison of autofocus methods for automated microscopy. , 1991, Cytometry.

[124]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[125]  David Salesin,et al.  Interactive digital photomontage , 2004, SIGGRAPH 2004.

[126]  Tien Tsin,et al.  Image Partial Blur Detection and Classification , 2013 .

[127]  Gregory Dudek,et al.  Image stitching with dynamic elements , 2009, Image Vis. Comput..

[128]  Anat Levin,et al.  Blind Motion Deblurring Using Image Statistics , 2006, NIPS.

[129]  Erik Reinhard,et al.  Progressive color transfer for images of arbitrary dynamic range , 2011, Comput. Graph..

[130]  Yasuyuki Matsushita,et al.  Smoothly varying affine stitching , 2011, CVPR 2011.

[131]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[132]  André Kaup,et al.  Histogram-Based Prefiltering for Luminance and Chrominance Compensation of Multiview Video , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[133]  François Pitié,et al.  Automated colour grading using colour distribution transfer , 2007, Comput. Vis. Image Underst..

[135]  David G. Lowe,et al.  Shape indexing using approximate nearest-neighbour search in high-dimensional spaces , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[136]  Lizhuang Ma,et al.  Gradient‐Preserving Color Transfer , 2009, Comput. Graph. Forum.

[137]  Stefan Winkler,et al.  A no-reference perceptual blur metric , 2002, Proceedings. International Conference on Image Processing.

[138]  Bill Triggs,et al.  Detecting Keypoints with Stable Position, Orientation, and Scale under Illumination Changes , 2004, ECCV.

[139]  Dani Lischinski,et al.  Gradient Domain High Dynamic Range Compression , 2023 .

[140]  Cordelia Schmid,et al.  Evaluation of Interest Point Detectors , 2000, International Journal of Computer Vision.

[141]  Marcelo H. Ang,et al.  Practical issues in pixel-based autofocusing for machine vision , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[142]  P. Jansson.  Deconvolution of images and spectra , 1997 .

[143]  Nicolas D. Georganas,et al.  Fast color correction using principal regions mapping in different color spaces , 2004, Real Time Imaging.

[144]  K. C. A. Smith,et al.  An automatic focusing and astigmatism correction system for the SEM and CTEM , 1982 .