Enhanced Soft 3D Reconstruction Method with an Iterative Matching Cost Update Using Object Surface Consensus

In this paper, we propose a multi-view stereo matching method, EnSoft3D (Enhanced Soft 3D Reconstruction) to obtain dense and high-quality depth images. Multi-view stereo is one of the high-interest research areas and has wide applications. Motivated by the Soft3D reconstruction method, we introduce a new multi-view stereo matching scheme. The original Soft3D method is introduced for novel view synthesis, while occlusion-aware depth is also reconstructed by integrating the matching costs of the Plane Sweep Stereo (PSS) and soft visibility volumes. However, the Soft3D method has an inherent limitation because the erroneous PSS matching costs are not updated. To overcome this limitation, the proposed scheme introduces an update process of the PSS matching costs. From the object surface consensus volume, an inverse consensus kernel is derived, and the PSS matching costs are iteratively updated using the kernel. The proposed EnSoft3D method reconstructs a highly accurate 3D depth image because both the multi-view matching cost and soft visibility are updated simultaneously. The performance of the proposed method is evaluated by using structured and unstructured benchmark datasets. Disparity error is measured to verify 3D reconstruction accuracy, and both PSNR and SSIM are measured to verify the simultaneous enhancement of view synthesis.

[1]  Lifeng Sun,et al.  Cross-Scale Cost Aggregation for Stereo Matching , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Lifeng Sun,et al.  Binary stereo matching , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[3]  Jean-Charles Bricola,et al.  Morphological processing of stereoscopic image superimpositions for disparity map estimation , 2016 .

[4]  Subhransu Maji,et al.  Multi-view Convolutional Neural Networks for 3D Shape Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[5]  Xiang Li,et al.  Accurate Dense Stereo Matching Based on Image Segmentation Using an Adaptive Multi-Cost Approach , 2016, Symmetry.

[6]  Michael S. Brown,et al.  SPM-BP: Sped-Up PatchMatch Belief Propagation for Continuous MRFs , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[7]  Jun Yu,et al.  Probability contour guided depth map inpainting and superresolution using non-local total generalized variation , 2018, Multimedia Tools and Applications.

[8]  Li Zhang,et al.  Soft 3D reconstruction for view synthesis , 2017, ACM Trans. Graph..

[9]  Williem,et al.  Deep self-guided cost aggregation for stereo matching , 2018, Pattern Recognit. Lett..

[10]  Stephen Lin,et al.  DPSNet: End-to-end Deep Plane Sweep Stereo , 2019, ICLR.

[11]  Michael Goesele,et al.  Multi-frame stereo matching with edges, planes, and superpixels , 2019, Image Vis. Comput..

[12]  Ding Yuan,et al.  Sparse representation over discriminative dictionary for stereo matching , 2017, Pattern Recognit..

[13]  Long Quan,et al.  Recurrent MVSNet for High-Resolution Multi-View Stereo Depth Inference , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Minglun Gong,et al.  Disparity Filtering with 3D Convolutional Neural Networks , 2018, 2018 15th Conference on Computer and Robot Vision (CRV).

[15]  Jing Xu,et al.  Point-Based Multi-View Stereo Network , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[16]  In-So Kweon,et al.  High-Quality Depth from Uncalibrated Small Motion Clip , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Truong Q. Nguyen,et al.  Efficient stereo-to-multiview synthesis , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[18]  H. Hirschmüller.  Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information , 2005, CVPR.

[19]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Konrad Schindler,et al.  Massively Parallel Multiview Stereopsis by Surface Normal Diffusion , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[21]  Mattia Rossi,et al.  DeepC-MVS: Deep Confidence Prediction for Multi-View Stereo Reconstruction , 2020, 2020 International Conference on 3D Vision (3DV).

[22]  Andrew W. Fitzgibbon,et al.  PMBP: PatchMatch Belief Propagation for Correspondence Field Estimation , 2014, International Journal of Computer Vision.

[23]  Ikuko Shimizu,et al.  High accuracy local stereo matching using DoG scale map , 2017, 2017 Fifteenth IAPR International Conference on Machine Vision Applications (MVA).

[24]  Pascal Fua,et al.  On benchmarking camera calibration and multi-view stereo for high resolution imagery , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[26]  Yong-Sheng Chen,et al.  Pyramid Stereo Matching Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Can Bal,et al.  On consistent inter-view synthesis for autostereoscopic displays , 2012 .

[29]  Qingxiong Yang,et al.  Stereo Matching Using Tree Filtering , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Shaojie Shen,et al.  MVDepthNet: Real-Time Multiview Depth Estimation Neural Network , 2018, 2018 International Conference on 3D Vision (3DV).

[31]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Heiko Hirschmüller,et al.  Evaluation of Cost Functions for Stereo Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[33]  Takeshi Naemura,et al.  Graph Cut Based Continuous Stereo Matching Using Locally Shared Labels , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Xiaoyan Hu,et al.  Least Commitment, Viewpoint-Based, Multi-view Stereo , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[35]  Marcelo Walter,et al.  An Artifact-Type Aware DIBR Method for View Synthesis , 2018, IEEE Signal Processing Letters.

[36]  Carsten Rother,et al.  PatchMatch Stereo - Stereo Matching with Slanted Support Windows , 2011, BMVC.

[37]  Ramin Zabih,et al.  Non-parametric Local Transforms for Computing Visual Correspondence , 1994, ECCV.

[38]  Paul Debevec,et al.  DeepView: View Synthesis With Learned Gradient Descent , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Tomás Pajdla,et al.  Multi-view reconstruction preserving weakly-supported surfaces , 2011, CVPR 2011.

[40]  Stefano Mattoccia,et al.  Guided Stereo Matching , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Jan-Michael Frahm,et al.  Pixelwise View Selection for Unstructured Multi-View Stereo , 2016, ECCV.

[42]  Radim Sara,et al.  Refinement of Surface Mesh for Accurate Multi-View Reconstruction , 2010, Int. J. Virtual Real..

[43]  Hong Zhu,et al.  Stereo Matching With Fusing Adaptive Support Weights , 2019, IEEE Access.

[44]  Narendra Ahuja,et al.  DeepMVS: Learning Multi-view Stereopsis , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[45]  Mohammed Ghanbari,et al.  The accuracy of PSNR in predicting video quality for different video scenes and frame rates , 2012, Telecommun. Syst..

[46]  Yann LeCun,et al.  Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches , 2015, J. Mach. Learn. Res..

[47]  KanadeT.,et al.  A Stereo Matching Algorithm with an Adaptive Window , 1994 .

[48]  Ziyue Huang,et al.  Stereo matching based on multi-scale fusion and multi-type support regions. , 2019, Journal of the Optical Society of America. A, Optics, image science, and vision.

[49]  M. Rupp,et al.  Multiview synthesis from stereo views , 2012, 2012 19th International Conference on Systems, Signals and Image Processing (IWSSIP).

[50]  Yuesheng Zhu,et al.  Foreground Removal Approach for Hole Filling in 3D Video and FVV Synthesis , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[51]  Weiming Dong,et al.  Segment-tree based cost aggregation for stereo matching with enhanced segmentation advantage , 2013, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[52]  Shan Lin,et al.  Plane Completion and Filtering for Multi-View Stereo Reconstruction , 2019, GCPR.

[53]  Richard Szeliski,et al.  High-accuracy stereo depth maps using structured light , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[54]  John Flynn,et al.  Stereo magnification , 2018, ACM Trans. Graph..

[55]  Jan-Michael Frahm,et al.  PatchMatch Based Joint View Selection and Depthmap Estimation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[56]  Takeshi Naemura,et al.  Continuous 3D Label Stereo Matching Using Local Expansion Moves , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  D. R. K. Brownrigg,et al.  The weighted median filter , 1984, CACM.

[58]  Gaochang Wu,et al.  Joint view synthesis and disparity refinement for stereo matching , 2019, Frontiers of Computer Science.

[59]  Radu Horaud,et al.  Topology-Adaptive Mesh Deformation for Surface Evolution, Morphing, and Multiview Reconstruction , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  Torsten Sattler,et al.  A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[61]  Enhua Wu,et al.  Constant Time Weighted Median Filtering for Stereo Matching and Beyond , 2013, 2013 IEEE International Conference on Computer Vision.

[62]  Haidi Ibrahim,et al.  Stereo matching algorithm based on per pixel difference adjustment, iterative guided filter and graph segmentation , 2017, J. Vis. Commun. Image Represent..

[63]  Carsten Rother,et al.  REal-time local stereo matching using guided image filtering , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[64]  Robert T. Collins,et al.  A space-sweep approach to true multi-image matching , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[65]  Graham Fyffe,et al.  Stereo Magnification: Learning View Synthesis using Multiplane Images , 2018, ArXiv.

[66]  Jonathan T. Barron,et al.  Pushing the Boundaries of View Extrapolation With Multiplane Images , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[67]  Li Zhang,et al.  PMSC: PatchMatch-Based Superpixel Cut for Accurate Stereo Matching , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[68]  Haidi Ibrahim,et al.  Improvement of stereo matching algorithm for 3D surface reconstruction , 2018, Signal Process. Image Commun..

[69]  Dongxiao Li,et al.  Fast stereo matching using adaptive guided filtering , 2014, Image Vis. Comput..