True multi-image alignment and its application to mosaicing and lens distortion correction

Multiple images of a scene are related through 2D/3D view transformations and linear and non-linear camera transformations. In all the traditional techniques to compute these transformations, especially the ones relying on direct intensity gradients, one image and its coordinate system have been assumed to be ideal and distortion free. In this paper, we present a formulation and an algorithm for true multi-image alignment that does not rely on the measurements of a reference image being distortion free. For instance, in the presence of lens distortion, none of the images can be assumed to be ideal. In our formulation, all the images are modeled as intensity measurements represented in their respective coordinate systems, each of which is related to an ideal coordinate system through an interior camera transformation and an exterior view transformation. The goal of the accompanying algorithm is to compute an image in the ideal coordinate system while solving for the transformations that relate the ideal system with each of the data images. Key advantages of the technique presented in this paper are: (i) no reliance on one distortion free image, (ii) ability to register images and compute coordinate transformations even when the multiple images are of an extended scene with no overlap between the first and last frame of the sequence, and (iii) ability to handle linear and non-linear transformations within the same framework. The new algorithm is evaluated in the context of two applications: (i) correction of lens distortion, and (ii) creation of video mosaics.

[1]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[2]  Keith J. Hanna,et al.  Combining stereo and motion analysis for direct estimation of scene structure , 1993, 1993 (4th) International Conference on Computer Vision.

[3]  Michal Irani,et al.  Representation of scenes from collections of images , 1995, Proceedings IEEE Workshop on Representation of Visual Scenes (In Conjunction with ICCV'95).

[4]  Harpreet S. Sawhney,et al.  Influence of global constraints and lens distortion on pose and appearance recovery from a purely rotating camera , 1998, Proceedings Fourth IEEE Workshop on Applications of Computer Vision. WACV'98 (Cat. No.98EX201).

[5]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[6]  Shmuel Peleg,et al.  Panoramic mosaics by manifold projection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  Michael J. Black,et al.  The robust estimation of multiple motions: Affine and piecewise smooth flow fields , 1993 .

[8]  Zhengyou Zhang,et al.  On the epipolar geometry between two images with lens distortion , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[9]  P. Anandan,et al.  Hierarchical Model-Based Motion Estimation , 1992, ECCV.

[10]  Richard Szeliski,et al.  Image mosaicing for tele-reality applications , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[11]  P. Anandan,et al.  Accurate computation of optical flow by using layered motion representations , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[12]  Harry Shum,et al.  Construction and refinement of panoramic mosaics with global and local alignment , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[13]  Michal Irani,et al.  Detecting and Tracking Multiple Moving Objects Using Temporal Integration , 1992, ECCV.

[14]  P. Anandan,et al.  Mosaic based representations of video sequences and their applications , 1995, Proceedings of IEEE International Conference on Computer Vision.

[15]  Harpreet S. Sawhney,et al.  Robust Video Mosaicing through Topology Inference and Local to Global Alignment , 1998, ECCV.

[16]  P. Anandan,et al.  Direct recovery of shape from multiple views: a parallax based approach , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[17]  Harpreet S. Sawhney,et al.  Layered representation of motion video using robust maximum-likelihood estimation of mixture models and MDL encoding , 1995, Proceedings of IEEE International Conference on Computer Vision.

[18]  Harpreet S. Sawhney,et al.  Model-based 2D&3D dominant motion estimation for mosaicing and video representation , 1995, Proceedings of IEEE International Conference on Computer Vision.

[19]  Richard I. Hartley,et al.  In Defense of the Eight-Point Algorithm , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Gideon P. Stein,et al.  Lens distortion calibration using point correspondences , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[21]  Harpreet S. Sawhney.  Simplifying motion and structure analysis using planar parallax and image warping , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[22]  Steve Mann,et al.  Virtual bellows: constructing high quality stills from video , 1994, Proceedings of 1st International Conference on Image Processing.

[23]  Y. Tsai Roger.  An Efficient and Accurate Camera Calibration Technique For 3D Machine Vision , 1986, CVPR 1986.