Techniques for Seamless Color Registration and Mapping on Dense 3D Models

Today’s most widely used 3D digitization approach is a combination of active geometric sensing, mainly using laser scanning, with active or passive color sensing, mostly using digital photography. Producing a seamless colored object, starting from a geometric representation and a set of photographs, is a data fusion problem requiring effective solutions for image-to-geometry registration, and color mapping and blending. This chapter provides a brief survey of the state-of-the-art solutions, ranging from manual approaches to fully scalable automated methods.

[1]  Daniel Cohen-Or,et al.  4-points congruent sets for robust pairwise surface registration , 2008, ACM Trans. Graph..

[2]  Adam Baumberg,et al.  Blending Images for Texturing 3D Models , 2002, BMVC.

[3]  Ruggero Pintus,et al.  Mont’e Scan , 2015, ACM Journal on Computing and Cultural Heritage.

[4]  Kok-Lim Low,et al.  Automatic registration of color images to 3D geometry , 2009, CGI.

[5]  Holly E. Rushmeier,et al.  The 3D Model Acquisition Pipeline , 2002, Comput. Graph. Forum.

[6]  Jan-Michael Frahm,et al.  3D model matching with Viewpoint-Invariant Patches (VIP) , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Fabio Remondino,et al.  Heritage Recording and 3D Modeling with Photogrammetry and 3D Scanning , 2011, Remote. Sens..

[8]  John W. Fisher,et al.  Automatic registration of LIDAR and optical images of urban scenes , 2009, CVPR.

[9]  Atsushi Nakazawa,et al.  The Great Buddha Project: Digitally Archiving, Restoring, and Analyzing Cultural Heritage Objects , 2007, International Journal of Computer Vision.

[10]  Paolo Cignoni,et al.  Acquiring, stitching and blending diffuse appearance attributes on 3D models , 2016, The Visual Computer.

[11]  J.-Angelo Beraldin,et al.  A framework for the registration of color images with 3D models , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[12]  Paolo Cignoni,et al.  Minimizing user intervention in registering 2D images to 3D models , 2005, The Visual Computer.

[13]  Derek Hoiem,et al.  Single-image shadow detection and removal using paired regions , 2011, CVPR 2011.

[14]  David Nister,et al.  Alignment of continuous video onto 3D point clouds , 2004, CVPR 2004.

[15]  Linda G. Shapiro,et al.  Acquisition and visualization of colored 3D objects , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[16]  Diego F. Nehab,et al.  Efficiently combining positions and normals for precise 3D geometry , 2005, SIGGRAPH 2005.

[17]  Shuchang Xu,et al.  Highlight Detection and Removal Based on Chromaticity , 2005, ICIAR.

[18]  Ruggero Pintus,et al.  A Fast and Robust Framework for Semiautomatic and Automatic Registration of Photographs to 3D Geometry , 2015, ACM Journal on Computing and Cultural Heritage.

[19]  Paolo Cignoni,et al.  Reconstructing Textured Meshes from Multiple Range RGB Maps , 2002, VMV.

[20]  A. Zakhor,et al.  Fast 3D model generation in urban environments , 2001, Conference Documentation International Conference on Multisensor Fusion and Integration for Intelligent Systems. MFI 2001 (Cat. No.01TH8590).

[21]  Michael Wimmer,et al.  Large-Scale Point-Cloud Visualization through Localized Textured Surface Reconstruction , 2014, IEEE Transactions on Visualization and Computer Graphics.

[22]  Marc Levoy.  The Digital Michelangelo Project , 1999, Comput. Graph. Forum.

[23]  Victor S. Lempitsky,et al.  Seamless Mosaicing of Image-Based Texture Maps , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  R. Pintus,et al.  A Streaming Framework for Seamless Detailed Photo Blending on Massive Point Clouds , 2011, Eurographics.

[25]  Szymon Rusinkiewicz,et al.  Merge2-3D: Combining Multiple Normal Maps with 3D Surfaces , 2014, 2014 2nd International Conference on 3D Vision.

[26]  Richard Szeliski,et al.  Using Force Fields Derived from 3D Distance Maps for Inferring the Attitude of a 3D Rigid Object , 1992, ECCV.

[27]  Holly E. Rushmeier,et al.  High-Quality Texture Reconstruction from Multiple Scans , 2001, IEEE Trans. Vis. Comput. Graph..

[28]  Katsushi Ikeuchi,et al.  Omnidirectional texturing based on robust 3D registration through Euclidean reconstruction from two spherical images , 2010, Comput. Vis. Image Underst..

[29]  Daniel Cohen-Or,et al.  Seamless Montage for Texturing Models , 2010, Comput. Graph. Forum.

[30]  Paolo Cignoni,et al.  Improved color acquisition and mapping on 3D models via flash-based photography , 2010, JOCCH.

[31]  Stephen Lin,et al.  A New In-Camera Imaging Model for Color Computer Vision and Its Application , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Ioannis Stamos,et al.  Automatic registration of 2-D with 3-D imagery in urban environments , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[33]  R.A. Johnston,et al.  Texture Mapping for the FastSCAN™ hand-held laser scanner , 2008, 2008 23rd International Conference Image and Vision Computing New Zealand.

[34]  Roberto Scopigno,et al.  A Statistical Method for SVBRDF Approximation from Video Sequences in General Lighting Conditions , 2012, Comput. Graph. Forum.

[35]  Konrad Klein,et al.  Texturing 3D Models of Real World Objects from Multiple Unregistered Photographic Views , 1999, Comput. Graph. Forum.

[36]  Marc Levoy,et al.  The digital Michelangelo project: 3D scanning of large statues , 2000, SIGGRAPH.

[37]  Eric Q. Li,et al.  A general texture mapping framework for image-based 3D modeling , 2010, 2010 IEEE International Conference on Image Processing.

[38]  Jean-Philippe Pons,et al.  Seamless image-based texture atlases using multi-band blending , 2008, 2008 19th International Conference on Pattern Recognition.

[39]  George Pavlidis,et al.  Multi-image 3D reconstruction data evaluation , 2014 .

[40]  Vítor Sequeira,et al.  3D reality modelling: photo-realistic 3D models of real world scenes , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[41]  Hans-Peter Seidel,et al.  A Silhouette-Based Algorithm for Texture Registration and Stitching , 2001, Graph. Model..

[42]  Dietmar Saupe,et al.  Highly-Automatic MI Based Multiple 2D/3D Image Registration Using Self-initialized Geodesic Feature Correspondences , 2009, ACCV.

[43]  Paul A. Viola,et al.  Alignment by Maximization of Mutual Information , 1997, International Journal of Computer Vision.

[44]  Dietmar Saupe,et al.  Stochastic Optimization of Multiple Texture Registration Using Mutual Information , 2007, DAGM-Symposium.

[45]  Anselmo Lastra,et al.  Automatic image alignment for 3D environment modeling , 2004, Proceedings. 17th Brazilian Symposium on Computer Graphics and Image Processing.

[46]  Ioannis Stamos,et al.  Automatic 3D to 2D registration for the photorealistic rendering of urban scenes , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[47]  Toyohisa Kaneko,et al.  Efficient and Handy Texture Mapping on 3D Surfaces , 1999, Comput. Graph. Forum.

[48]  Szymon Rusinkiewicz,et al.  Estimating the Laplace‐Beltrami Operator by Restricting 3D Functions , 2009, Comput. Graph. Forum.

[49]  Jan-Michael Frahm,et al.  Building Rome on a Cloudless Day , 2010, ECCV.

[50]  Richard Szeliski,et al.  Alignment of 3D point clouds to overhead images , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[51]  Paolo Cignoni,et al.  Masked photo blending: Mapping dense photographic data set on high-resolution sampled 3D models , 2008, Comput. Graph..

[52]  Thomas Malzbender,et al.  Polynomial texture maps , 2001, SIGGRAPH.

[53]  Anselmo Lastra,et al.  Metrics and Optimization Techniques for Registration of Color to Laser Range Scans , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[54]  Christian Früh,et al.  Constructing 3D City Models by Merging Aerial and Ground Views , 2003, IEEE Computer Graphics and Applications.

[55]  Hans-Peter Seidel,et al.  Relighting objects from image collections , 2009, CVPR 2009.

[56]  Ruggero Pintus,et al.  Fast and Robust Semi-Automatic Registration of Photographs to 3D Geometry , 2011, VAST.

[57]  Ioannis Stamos,et al.  Integrating Automated Range Registration with Multiview Geometry for the Photorealistic Modeling of Large-Scale Scenes , 2008, International Journal of Computer Vision.

[58]  Michael Goesele,et al.  Let There Be Color! Large-Scale Texturing of 3D Reconstructions , 2014, ECCV.

[59]  Paolo Cignoni,et al.  Flow-Based Local Optimization for Image-to-Geometry Projection , 2012, IEEE Transactions on Visualization and Computer Graphics.

[60]  Roberto Scopigno,et al.  Image‐to‐Geometry Registration: a Mutual Information Method exploiting Illumination‐related Geometric Properties , 2009, Comput. Graph. Forum.

[61]  Paolo Cignoni,et al.  Stereo Light Probe , 2008, Comput. Graph. Forum.

[62]  Roberto Scopigno,et al.  Mapping Highly Detailed Colour Information on Extremely Dense 3D Models: The Case of David's Restoration , 2008, Comput. Graph. Forum.

[63]  Roberto Scopigno,et al.  Fully Automatic Registration of Image Sets on Approximate Geometry , 2012, International Journal of Computer Vision.

[64]  Roberto Scopigno,et al.  Global refinement of image-to-geometry registration for color projection , 2013, 2013 Digital Heritage International Congress (DigitalHeritage).

[65]  Charles V. Stewart,et al.  Estimating the Location of a Camera with Respect to a 3D Model , 2007, Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007).

[66]  Hans-Peter Seidel,et al.  Automated texture registration and stitching for real world models , 2000, Proceedings the Eighth Pacific Conference on Computer Graphics and Applications.

[67]  Marc Pollefeys,et al.  Interactive 3D architectural modeling from unordered photo collections , 2008, SIGGRAPH 2008.

[68]  Ruggero Pintus,et al.  Improving the digitization of shape and color of 3D artworks in a cluttered environment , 2013, 2013 Digital Heritage International Congress (DigitalHeritage).

[69]  David G. Lowe,et al.  Fitting Parameterized Three-Dimensional Models to Images , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[70]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[71]  George Wolberg,et al.  Multiview Geometry for Texture Mapping 2D Images Onto 3D Range Data , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[72]  Roberto Scopigno,et al.  Removing Shadows for Color Projection Using Sun Position Estimation , 2010, VAST.

[73]  Ruggero Pintus,et al.  Fast low-memory seamless photo blending on massive point clouds using a streaming framework , 2011, JOCCH.

[74]  Daniel Cremers,et al.  Superresolution texture maps for multiview reconstruction , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[75]  Vladlen Koltun,et al.  Color map optimization for 3D reconstruction with consumer depth cameras , 2014, ACM Trans. Graph..

[76]  Renato Pajarola,et al.  SOAR: Stochastic Optimization for Affine global point set Registration , 2014, VMV.