3D panoramic imaging for virtual environment construction

The project is concerned with the development of algorithms for the creation of photo-realistic 3D virtual environments, overcoming problems in mosaicing, colour and lighting changes, correspondence search speed and correspondence errors due to lack of surface texture. A number of related new algorithms have been investigated for image stitching, content based colour correction and efficient 3D surface reconstruction. All of the investigations were undertaken by using multiple views from normal digital cameras, web cameras and a ”one-shot” panoramic system. In the process of 3D reconstruction a new interest points based mosaicing method, a new interest points based colour correction method, a new hybrid feature and area based correspondence constraint and a new structured light based 3D reconstruction method have been investigated. The major contributions and results can be summarised as follows: • A new interest point based image stitching method has been proposed and investigated. The robustness of interest points has been tested and evaluated. Interest points have been proved robust to changes in lighting, viewpoint, rotation and scale. • A new interest point based method for colour correction has been proposed and investigated. The results of linear and linear plus affine colour transforms have proved more accurate than traditional diagonal transforms in accurately matching colours in panoramic images. • A new structured light based method for correspondence point based 3D reconstruction has been proposed and investigated. The method has been proved to increase the accuracy of the correspondence search for areas with low texture. Correspondence speed has also been increased with a new hybrid feature and area based correspondence search constraint. • Based on the investigation, a software framework has been developed for image based 3D virtual environment construction. The GUI includes abilities for importing images, colour correction, mosaicing, 3D surface reconstruction, texture recovery and visualisation. • 11 research papers have been published.

[1]  Steve Mann,et al.  Virtual bellows: constructing high quality stills from video , 1994, Proceedings of 1st International Conference on Image Processing.

[2]  Tom,et al.  Central Panoramic Cameras: Design and Geometry Central Panoramic Cameras: Design and Geometry Tomm a S Svoboda, Tomm a S Pajdla and Vv Aclav Hlavv a C , 1998 .

[3]  Laurent Moll,et al.  Real time correlation-based stereo: algorithm, implementations and applications , 1993 .

[4]  Hyun-Chul Kim,et al.  Stitching reliability for estimating camera focal length in panoramic image mosaicing , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[5]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[6]  Liang Zhang.  HIERARCHICAL BLOCK-BASED DISPARITY ESTIMATION USING MEAN ABSOLUTE DIFFERENCE AND DYNAMIC PROGRAMMING , 2001 .

[7]  Aly A. Farag,et al.  Integrating stereo and shape from shading , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[8]  Yael Pritch,et al.  Optics for Omnistereo Imaging , 2001 .

[9]  Fadi Dornaika,et al.  Binocular matching constraints from motion , 2002, Pattern Recognit..

[10]  Aggelos K. Katsaggelos,et al.  Dense Disparity Estimation with a Divide-and-Conquer Disparity Space Image Technique , 1999, IEEE Trans. Multim..

[11]  Yoichi Muraoka,et al.  Real-time image mosaicing from a video sequence , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[12]  Shenchang Eric Chen,et al.  QuickTime VR: an image-based approach to virtual environment navigation , 1995, SIGGRAPH.

[13]  Cornelius Malerczyk,et al.  3D Reconstruction of Sports Events for Digital TV , 2003, WSCG.

[14]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[15]  Narendra Ahuja,et al.  Surfaces from Stereo: Integrating Feature Matching, Disparity Estimation, and Contour Detection , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  João M. F. Xavier,et al.  Multi-view correspondence by enforcement of rigidity constraints , 2007, Image Vis. Comput..

[17]  Alan M. McIvor,et al.  Accurate 3D measurement using a structured light system , 1998, Image Vis. Comput..

[18]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[19]  H. Knutsson,et al.  Estimating Multiple Depths in Semi-transparent Stereo Images , 1999 .

[20]  Takeo Kanade,et al.  A Cooperative Algorithm for Stereo Matching and Occlusion Detection , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Harry Shum,et al.  Omnivergent Stereo , 2004, International Journal of Computer Vision.

[22]  Cordelia Schmid,et al.  Indexing Based on Scale Invariant Interest Points , 2001, ICCV.

[23]  Andrew P. Witkin,et al.  Scale-Space Filtering , 1983, IJCAI.

[24]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Quang-Tuan Luong,et al.  Self-Calibration of a Moving Camera from Point Correspondences and Fundamental Matrices , 1997, International Journal of Computer Vision.

[26]  Roberto Cipolla,et al.  Structure and motion from silhouettes , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[27]  Reinhold Ritter,et al.  3D deformation measurement in small areas based on grating method and photogrammetry , 1996, Other Conferences.

[28]  G. Medioni,et al.  Tensor Voting : Theory and Applications , 2000 .

[29]  B. S. Manjunath,et al.  Panoramic video capturing and compressed domain virtual camera control , 2001, MULTIMEDIA '01.

[30]  Jong-Il Park,et al.  Hierarchical Depth Mapping from Multiple Cameras , 1997, ICIAP.

[31]  Gerhard Paar,et al.  The Beagle 2 stereo camera system , 2005 .

[32]  Cordelia Schmid,et al.  Comparing and evaluating interest points , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[33]  Richard Szeliski,et al.  Creating full view panoramic image mosaics and environment maps , 1997, SIGGRAPH.

[34]  B. Funt,et al.  Diagonal versus affine transformations for color correction. , 2000, Journal of the Optical Society of America. A, Optics, image science, and vision.

[35]  Li Zhao,et al.  Mosaic image method: a local and global method , 1999, Pattern Recognit..

[36]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[37]  Stephen M. Smith,et al.  SUSAN—A New Approach to Low Level Image Processing , 1997, International Journal of Computer Vision.

[38]  Shree K. Nayar,et al.  Real-Time Omnidirectional and Panoramic Stereo , 1998 .

[39]  Aly A. Farag,et al.  Fast 3D object reconstruction using trinocular vision and structured light , 1998, Other Conferences.

[40]  Nikolaos Grammalidis,et al.  Disparity and occlusion estimation in multiocular systems and their coding for the communication of multiview image sequences , 1998, IEEE Trans. Circuits Syst. Video Technol..

[41]  Rachid Deriche,et al.  Using geometric corners to build a 2D mosaic from a set of images , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[42]  Leonard McMillan,et al.  Acquiring Immersive Virtual Environments with an Uncalibrated Camera , 1995 .

[43]  Charles V. Stewart,et al.  Robust hierarchical algorithm for constructing a mosaic from images of the curved human retina , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[44]  Allen R. Hanson,et al.  STEREO MOSAICS FROM A MOVING VIDEO CAMERA FOR ENVIRONMENTAL MONITORING , 1999 .

[45]  Takeo Kanade,et al.  A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  KanadeTakeo,et al.  Shape and motion from image streams under orthography , 1992 .

[47]  Maurizio Pilu,et al.  A direct method for stereo correspondence based on singular value decomposition , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[48]  Rachid Deriche,et al.  A Robust Technique for Matching two Uncalibrated Images Through the Recovery of the Unknown Epipolar Geometry , 1995, Artif. Intell..

[49]  Naokazu Yokoya,et al.  Telepresence by Real-Time View-Dependent Image Generation from Omnidirectional Video Streams , 1998, Comput. Vis. Image Underst..

[50]  Shmuel Peleg,et al.  Stereo panorama with a single camera , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[51]  Gui Yun Tian,et al.  Panoramic imaging - a review , 2003, Comput. Graph..

[52]  Michael H. Brill,et al.  The relation between the color of the illuminant and the color of the illuminated object , 1995 .

[53]  Emanuele Trucco,et al.  Efficient stereo with multiple windowing , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[54]  Harry Shum,et al.  Review of image-based rendering techniques , 2000, Visual Communications and Image Processing.

[55]  Harry Shum,et al.  Construction and refinement of panoramic mosaics with global and local alignment , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[56]  Steven M. Seitz,et al.  Physically-valid view synthesis by image interpolation , 1995, Proceedings IEEE Workshop on Representation of Visual Scenes (In Conjunction with ICCV'95).

[57]  Yi-Ping Hung,et al.  Panoramic Stereo Imaging System with Automatic Disparity Warping and Seaming , 1998, Graph. Model. Image Process..

[58]  Narendra Ahuja,et al.  High Dynamic Range Panoramic Imaging , 2001, ICCV.

[59]  Jake K. Aggarwal,et al.  Structure from stereo-a review , 1989, IEEE Trans. Syst. Man Cybern..

[60]  Alexandru Tupan,et al.  Triangulation , 1997, Comput. Vis. Image Underst..

[61]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[62]  Boaz J. Super,et al.  Patch-Based Stereo in a General Binocular Viewing Geometry , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[63]  Gregory Dudek,et al.  Automated creation of image-based virtual reality , 1997, Other Conferences.

[64]  Daniel G. Aliaga,et al.  Plenoptic stitching: a scalable method for reconstructing 3D interactive walk throughs , 2001, SIGGRAPH.

[65]  Steven M. Seitz,et al.  The Space of All Stereo Images , 2004, International Journal of Computer Vision.

[66]  Yoshiaki Shirai,et al.  Real-time omnidirectional stereo for obstacle detection and tracking in dynamic environments , 2001, Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the the Next Millennium (Cat. No.01CH37180).

[67]  Andreas Koschan,et al.  3D reconstruction of road surfaces using an integrated multi-sensory approach , 2007 .

[68]  Thomas O. Binford,et al.  Depth from Edge and Intensity Based Stereo , 1981, IJCAI.

[69]  Frédo Durand,et al.  A gentle introduction to bilateral filtering and its applications , 2007, SIGGRAPH Courses.

[70]  Narendra Ahuja,et al.  Motion and Structure From Two Perspective Views: Algorithms, Error Analysis, and Error Estimation , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[71]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[72]  Cordelia Schmid,et al.  Local Grayvalue Invariants for Image Retrieval , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[73]  Tomas Pajdla,et al.  Panoramic cameras for 3D computation , 2000 .

[74]  Hiroshi Ishiguro,et al.  Omni-Directional Stereo , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[75]  Chaur-Heh Hsieh,et al.  An Efficient Motion Estimation Algorithm for Video Coding Using Kalman Filter , 2002, Real Time Imaging.

[76]  H. C. Longuet-Higgins,et al.  A computer algorithm for reconstructing a scene from two projections , 1981, Nature.

[77]  M. S. Drew,et al.  Color constancy - Generalized diagonal transforms suffice , 1994 .

[78]  José M. F. Moura,et al.  Three-dimensional modeling from two-dimensional video , 2001, IEEE Trans. Image Process..

[79]  G. Healey,et al.  Global color constancy: recognition of objects by use of illumination-invariant properties of color distributions , 1994 .

[80]  Dan Xu,et al.  Complex wavelet-based image mosaics using edge-preserving visual perception modeling , 1999, Comput. Graph..

[81]  S. P. Mudur,et al.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[82]  Janne Heikkilä,et al.  Geometric Camera Calibration Using Circular Control Points , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[83]  Subhasis Chaudhuri,et al.  Optimal recovery of depth from defocused images using an MRF model , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[84]  Hans P. Moravec.  Rover Visual Obstacle Avoidance , 1981, IJCAI.

[85]  Nikolaos Grammalidis,et al.  Disparity field and depth map coding for multiview 3D image generation , 1998, Signal Process. Image Commun..

[86]  Hiroyuki Ohno,et al.  An optical see-through display for mutual occlusion with a real-time stereovision system , 2001, Comput. Graph..

[87]  Ebroul Izquierdo,et al.  Image Analysis for 3D Modeling, Rendering, and Virtual View Generation , 1998, Comput. Vis. Image Underst..

[88]  Richard Szeliski,et al.  Video mosaics for virtual environments , 1996, IEEE Computer Graphics and Applications.

[89]  Olivier D. Faugeras,et al.  The fundamental matrix: Theory, algorithms, and stability analysis , 2004, International Journal of Computer Vision.

[90]  Gui Yun Tian,et al.  Comprehensive interest points based imaging mosaic , 2003, Pattern Recognit. Lett..

[91]  E. Land,et al.  Lightness and retinex theory. , 1971, Journal of the Optical Society of America.

[92]  Kin Hong Wong,et al.  Panoramic Video Representation using Mosaic Image , 1999 .

[93]  Olga Veksler,et al.  Stereo Correspondence with Compact Windows via Minimum Ratio Cycle , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[94]  Camillo J. Taylor,et al.  Reconstruction of Linearly Parameterized Models from Single Images with a Camera of Unknown Focal Length , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[95]  Cordelia Schmid,et al.  Evaluation of Interest Point Detectors , 2000, International Journal of Computer Vision.

[96]  Harry Shum,et al.  Stereo reconstruction from multiperspective panoramas , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[97]  Changming Sun,et al.  Fast Stereo Matching Using Rectangular Subregioning and 3D Maximum-Surface Techniques , 2002, International Journal of Computer Vision.

[98]  Ramesh C. Jain,et al.  Structure from motion-a critical analysis of methods , 1991, IEEE Trans. Syst. Man Cybern..

[99]  Nadia Magnenat-Thalmann,et al.  Generating Animatable 3D Virtual Humans from Photographs , 2000, Comput. Graph. Forum.

[100]  Richard Szeliski,et al.  Image mosaicing for tele-reality applications , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[101]  G. Finlayson,et al.  Colour indexing across illumination , 1999 .

[102]  Lisa M. Brown,et al.  A survey of image registration techniques , 1992, CSUR.

[103]  Naoya Ohta,et al.  Accuracy bounds and optimal computation of homography for image mosaicing applications , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.