CNLPA-MVS: Coarse-Hypotheses Guided Non-Local PatchMatch Multi-View Stereo

In multi-view stereo, unreliable matching in low-textured regions has a negative impact on the completeness of reconstructed models. Since the photometric consistency of low-textured regions is not discriminative under a local window, non-local information provided by the Markov Random Field (MRF) model can alleviate the matching ambiguity but limited in continuous space with high computational complexity. Owing to its sampling and propagation strategy, PatchMatch multi-view stereo methods have advantages in terms of optimizing continuous labeling problem. In this paper, we propose a novel method to address this problem, namely the Coarsehypotheses guided Non-Local PAtchMatch Multi-View Stereo (CNLPA-MVS), which takes the advantages of both MRF-based non-local methods and PatchMatch multi-view stereo and compensates for their defects mutually. First, we combined dynamic programing (DP) and sequential propagation along scanlines in parallel to perform CNLPA-MVS, thereby obtaining the optimal depth and normal hypotheses. Second, we introduced coarse inence within a universal window provided by winner-takes-all to eliminate the stripe artifacts caused by DP and improve completeness. Third, we added a local consistency strategy based on the hypotheses of similar color pixels sharing approximate values into CNLPA-MVS for further improving completeness. CNLPA-MVS was validated on public benchmarks and achieved state-ofthe-art performance with high completeness.

[1]  Carlos Hernandez,et al.  Multi-View Stereo: A Tutorial , 2015, Found. Trends Comput. Graph. Vis..

[2]  Torsten Sattler,et al.  A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Konrad Schindler,et al.  Massively Parallel Multiview Stereopsis by Surface Normal Diffusion , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[4]  Carsten Rother,et al.  PatchMatch Stereo - Stereo Matching with Slanted Support Windows , 2011, BMVC.

[5]  Davide Scaramuzza,et al.  Air-ground localization and map augmentation using monocular dense reconstruction , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[6]  Michael J. Black,et al.  Towards Probabilistic Volumetric Reconstruction Using Ray Potentials , 2015, 2015 International Conference on 3D Vision.

[7]  Pascal Fua,et al.  On benchmarking camera calibration and multi-view stereo for high resolution imagery , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[10]  Wenbing Tao,et al.  Multi-Scale Geometric Consistency Guided Multi-View Stereo , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Takeshi Naemura,et al.  Graph Cut Based Continuous Stereo Matching Using Locally Shared Labels , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Ying Wang,et al.  MARMVS: Matching Ambiguity Reduced Multiple View Stereo for Efficient Large Scale Scene Reconstruction , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Tao Guan,et al.  P-MVSNet: Learning Patch-Wise Matching Confidence Aggregation for Multi-View Stereo , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[14]  Jan-Michael Frahm,et al.  Pixelwise View Selection for Unstructured Multi-View Stereo , 2016, ECCV.

[15]  Changsheng Xu,et al.  Enhanced 3-D Modeling for Landmark Image Classification , 2012, IEEE Transactions on Multimedia.

[16]  Roberto Cipolla,et al.  Using Multiple Hypotheses to Improve Depth-Maps for Multi-View Stereo , 2008, ECCV.

[17]  Koichi Ogawara.  Approximate Belief Propagation by Hierarchical Averaging of Outgoing Messages , 2010, 2010 20th International Conference on Pattern Recognition.

[18]  Jan-Michael Frahm,et al.  PatchMatch Based Joint View Selection and Depthmap Estimation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Luc Van Gool,et al.  Progressive Prioritized Multi-view Stereo , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Tianli Yu,et al.  Efficient Message Representations for Belief Propagation , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[21]  Jie Liao,et al.  Pyramid Multi‐View Stereo with Local Consistency , 2019, Comput. Graph. Forum.

[22]  Alois Knoll,et al.  PM-Huber: PatchMatch with Huber Regularization for Stereo Matching , 2013, 2013 IEEE International Conference on Computer Vision.

[23]  Andrew W. Fitzgibbon,et al.  PMBP: PatchMatch Belief Propagation for Correspondence Field Estimation , 2014, International Journal of Computer Vision.

[24]  Qingshan Xu,et al.  Planar Prior Assisted PatchMatch Multi-View Stereo , 2019, AAAI.

[25]  Narendra Ahuja,et al.  DeepMVS: Learning Multi-view Stereopsis , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[27]  Andrew W. Fitzgibbon,et al.  Global stereo reconstruction under second order smoothness priors , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[29]  Roberto Cipolla,et al.  Multiview Stereo via Volumetric Graph-Cuts and Occlusion Robust Photo-Consistency , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Long Quan,et al.  MVSNet: Depth Inference for Unstructured Multi-view Stereo , 2018, ECCV.

[31]  Dani Lischinski,et al.  Joint bilateral upsampling , 2007, ACM Trans. Graph..

[32]  Vladimir Kolmogorov,et al.  Computing visual correspondence with occlusions using graph cuts , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[33]  Hendrik P. A. Lensch,et al.  Multi-View Depth Map Estimation With Cross-View Consistency , 2014, BMVC.

[34]  Jean-Philippe Pons,et al.  High Accuracy and Visibility-Consistent Dense Multiview Stereo , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Li Zhang,et al.  PMSC: PatchMatch-Based Superpixel Cut for Accurate Stereo Matching , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[36]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[37]  Michael S. Brown,et al.  SPM-BP: Sped-Up PatchMatch Belief Propagation for Continuous MRFs , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[38]  Andreas Klaus,et al.  Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure , 2006, 18th International Conference on Pattern Recognition (ICPR'06).