Multi-view stereo via depth map fusion: A coordinate decent optimization method

Multi-view stereo (MVS) plays a critical role in many practically important vision applications. Among the existing MVS methods, one typical approach is to fuse the depth maps from different views via minimization of the energy functional. However, these methods usually have expensive computational cost and are inflexible for extending to large neighborhood, leading to long run time and reconstruction artifacts. In this work, we propose a simple, efficient and flexible depth-map-fusion-based MVS reconstruction method: CoD-Fusion. The core idea of the method is to minimize the anisotropic or isotropic TV+L1 energy functional using the coordinate decent (CoD) algorithm. CoD performs TV+L1 minimization via solving a serial of voxel-wise L1 minimization sub-problems which can be efficiently solved using fast weighted median filtering (WMF). We then extend WMF to larger neighborhood to suppress reconstruction artifacts. The results of quantitative and qualitative evaluation validate the flexibility and efficiency of CoD-Fusion as a promising method for large scale MVS reconstruction.

[1]  Gonzalo R. Arce,et al.  A fast weighted median algorithm based on Quickselect , 2010, 2010 IEEE International Conference on Image Processing.

[2]  Derek Bradley,et al.  Markerless garment capture , 2008, SIGGRAPH 2008.

[3]  Jean Ponce,et al.  Modeling 3D Objects from Stereo Views and Recognizing Them in Photographs , 2006, ECCV.

[4]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[5]  Daniel Cremers,et al.  Continuous Global Optimization in Multiview 3D Reconstruction , 2007, International Journal of Computer Vision.

[6]  Daniel Cremers,et al.  Anisotropic Minimal Surfaces Integrating Photoconsistency and Normal Information for Multiview Stereo , 2010, ECCV.

[7]  Pascal Fua,et al.  Efficient large-scale multi-view stereo for ultra high-resolution image sets , 2011, Machine Vision and Applications.

[8]  Mumin Song,et al.  Overview of three-dimensional shape measurement using optical methods , 2000 .

[9]  Qionghai Dai,et al.  Continuous depth estimation for multi-view stereo , 2009, CVPR.

[10]  Thomas Brox,et al.  Point-Based 3D Reconstruction of Thin Objects , 2013, 2013 IEEE International Conference on Computer Vision.

[11]  Daniel Cremers,et al.  Multiview Stereo and Silhouette Consistency via Convex Functionals over Convex Domains , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Ilya Kostrikov,et al.  Probabilistic Labeling Cost for High-Accuracy Multi-view Reconstruction , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[14]  Zhanyi Hu,et al.  How to Select Good Neighboring Images in Depth-Map Merging Based 3D Modeling , 2014, IEEE Transactions on Image Processing.

[15]  Joachim Weickert,et al.  Anisotropic Range Image Integration , 2012, DAGM/OAGM Symposium.

[16]  Mila Nikolova,et al.  Algorithms for Finding Global Minimizers of Image Segmentation and Denoising Models , 2006, SIAM J. Appl. Math..

[17]  Lyndon Smith,et al.  Overview of passive and active vision techniques for hand-held 3D data acquistion , 2003, SPIE OPTO-Ireland.

[18]  Pau Gargallo,et al.  Minimizing the Reprojection Error in Surface Reconstruction from Images , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[19]  Andreas Geiger,et al.  Efficient Large-Scale Stereo Matching , 2010, ACCV.

[20]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[21]  Henrik Aanæs,et al.  Large Scale Multi-view Stereopsis Evaluation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  M. Nikolova An Algorithm for Total Variation Minimization and Applications , 2004 .

[23]  Leif Kobbelt,et al.  A Surface-Growing Approach to Multi-View Stereo Reconstruction , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Pascal Fua,et al.  On benchmarking camera calibration and multi-view stereo for high resolution imagery , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Jean-Philippe Pons,et al.  Towards high-resolution large-scale multi-view stereo , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Derek Bradley,et al.  Accurate multi-view reconstruction using robust binocular stereo and surface meshing , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Francis Schmitt,et al.  Silhouette and stereo fusion for 3D object modeling , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[28]  Eric Q. Li,et al.  Bundled depth-map merging for multi-view stereo , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[29]  Horst Bischof,et al.  Online 3D reconstruction using convex optimization , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[30]  Qingxiong Yang,et al.  A non-local cost aggregation method for stereo matching , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Horst Bischof,et al.  A Globally Optimal Algorithm for Robust TV-L1 Range Image Integration , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[32]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[33]  Shuhan Shen,et al.  Accurate Multiple View 3D Reconstruction Using Patch-Based Stereo for Large-Scale Scenes , 2013, IEEE Transactions on Image Processing.

[34]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[35]  Jan Boehm,et al.  Image selection in photogrammetric multi-view stereo methods for metric and complete 3D reconstruction , 2013, Optical Metrology.

[36]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  A. Verri,et al.  A compact algorithm for rectification of stereo pairs , 2000 .

[38]  Francis Schmitt,et al.  Silhouette and stereo fusion for 3D object modeling , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[39]  Roberto Cipolla,et al.  Multiview Stereo via Volumetric Graph-Cuts and Occlusion Robust Photo-Consistency , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Lei Zhang,et al.  A Cyclic Weighted Median Method for L1 Low-Rank Matrix Factorization with Missing Entries , 2013, AAAI.

[41]  Mubarak Shah,et al.  3D Model based Object Class Detection in An Arbitrary View , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[42]  M. Goesele,et al.  Fusion of depth maps with multiple scales , 2011, ACM Trans. Graph..

[43]  Roberto Cipolla,et al.  Using Multiple Hypotheses to Improve Depth-Maps for Multi-View Stereo , 2008, ECCV.

[44]  Daniel Cremers,et al.  An Experimental Comparison of Discrete and Continuous Shape Optimization Methods , 2008, ECCV.

[45]  Jules Bloomenthal,et al.  Polygonization of implicit surfaces , 1988, Comput. Aided Geom. Des..

[46]  A. Chambolle Practical, Unified, Motion and Missing Data Treatment in Degraded Video , 2004, Journal of Mathematical Imaging and Vision.

[47]  Marc Pollefeys,et al.  Multi-View Stereo via Graph Cuts on the Dual of an Adaptive Tetrahedral Mesh , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[48]  Luc Van Gool,et al.  Overlapping camera clustering through dominant sets for scalable 3D reconstruction , 2013, BMVC.

[49]  Michael M. Kazhdan,et al.  Poisson surface reconstruction , 2006, SGP '06.

[50]  Heiko Hirschmüller,et al.  Multi-Resolution Range Data Fusion for Multi-View Stereo Reconstruction , 2013, GCPR.

[51]  Heinrich Müller,et al.  Improved Laplacian Smoothing of Noisy Surface Meshes , 1999, Comput. Graph. Forum.

[52]  Oleg V. Michailovich,et al.  An Iterative Shrinkage Approach to Total-Variation Image Restoration , 2009, IEEE Transactions on Image Processing.

[53]  Jean-Philippe Pons,et al.  Minimizing the Multi-view Stereo Reprojection Error for Triangular Surface Meshes , 2008, BMVC.

[54]  S. Osher,et al.  A new median formula with applications to PDE based denoising , 2009 .

[55]  Zhiyong Feng,et al.  Robust visual tracking via online multiple instance learning with Fisher information , 2015, Pattern Recognit..

[56]  Carsten Rother,et al.  Fast cost-volume filtering for visual correspondence and beyond , 2011, CVPR 2011.

[57]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Vladimir Kolmogorov,et al.  Computing geodesics and minimal surfaces via graph cuts , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[59]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[60]  Jan-Michael Frahm,et al.  Real-Time Visibility-Based Fusion of Depth Maps , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[61]  Olivier D. Faugeras,et al.  Multi-View Stereo Reconstruction and Scene Flow Estimation with a Global Image-Based Matching Score , 2007, International Journal of Computer Vision.

[62]  Florent Lafarge,et al.  Hybrid multi-view reconstruction by Jump-Diffusion , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[63]  Kurt Konolige,et al.  Small Vision Systems: Hardware and Implementation , 1998 .

[64]  Xavier Bresson,et al.  Fast Global Minimization of the Active Contour/Snake Model , 2007, Journal of Mathematical Imaging and Vision.

[65]  Qionghai Dai,et al.  Noisy Depth Maps Fusion for Multiview Stereo Via Matrix Completion , 2012, IEEE Journal of Selected Topics in Signal Processing.

[66]  Marc Pollefeys,et al.  Turning Mobile Phones into 3D Scanners , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[67]  Radu Horaud,et al.  TransforMesh : A Topology-Adaptive Mesh-Based Approach to Surface Evolution , 2007, ACCV.

[68]  Victor S. Lempitsky,et al.  Oriented Visibility for Multiview Reconstruction , 2006, ECCV.