Three-dimensional rapid registration and reconstruction of multi-view rigid objects based on end-to-end deep surface model

Three-dimensional object reconstruction from multi-view images is an important topic in computer vision, which has attracted enormous attention during the past decades. With the further study in deep learning, remarkable progress of three-dimensional object reconstruct has been obtained in recent years. In this paper, we proposed three-dimensional rapid registration and reconstruction of multi-view rigid objects based on end-to-end deep surface model in the field of three-dimensional object reconstruction. Firstly, we introduce a matching algorithm called local stereo matching algorithm based on improved census transform and multi-scale spatial, aiming to improve the matching results for those regions. In cost aggregation step, guided map filtering algorithm with excellent gradient preserving property is introduced into Gaussian pyramid structure and regularization is added to strengthen cost volume consistency. Secondly, the improved inception RESNET module is added to improve the feature extraction ability of the network, and multiple features are extracted by using multiple network structures, and finally multiple features are sequentially input into the VRNN module to enhance the reconstruction effect of multi-view images. The experimental results show that our proposed method can not only achieve better reconstruction results, but also reconstruct more details and spend less time in training.

[1]  Pengjiang Qian,et al.  Seizure Classification From EEG Signals Using Transfer Learning, Semi-Supervised Learning and TSK Fuzzy System , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[2]  Ann-Shyn Chiang,et al.  Soma Detection in 3D Images of Neurons using Machine Learning Technique , 2017, Neuroinformatics.

[3]  Sunghee Choi,et al.  The power crust, unions of balls, and the medial axis transform , 2001, Comput. Geom..

[4]  Jiaming Liu,et al.  AMD3100 inhibits the migration and differentiation of neural stem cells after spinal cord injury , 2017, Scientific Reports.

[5]  Igor Skrjanc,et al.  Editorial A Successful Change From TNN to TNNLS and a Very Successful Year , 2013, IEEE Trans. Neural Networks Learn. Syst..

[6]  Chang-Dong Wang,et al.  Multi-view Intact Space Clustering , 2017, 2017 4th IAPR Asian Conference on Pattern Recognition (ACPR).

[7]  Sebastian Thrun,et al.  Upsampling range data in dynamic environments , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Pengtao Jia,et al.  Nonuniform multiview color texture mapping of image sequence and three-dimensional model for faded cultural relics with sift feature points , 2018 .

[9]  Renmin Han,et al.  DLBI: deep learning guided Bayesian inference for structure reconstruction of super-resolution fluorescence microscopy , 2018, Bioinform..

[10]  Longin Jan Latecki,et al.  Automatic Ensemble Diffusion for 3D Shape and Image Retrieval , 2019, IEEE Transactions on Image Processing.

[11]  Pengjiang Qian,et al.  Cluster Prototypes and Fuzzy Memberships Jointly Leveraged Cross-Domain Maximum Entropy Clustering , 2016, IEEE Transactions on Cybernetics.

[12]  Shengzhe Wang,et al.  Bayesian Framework with Non-local and Low-rank Constraint for Image Reconstruction , 2017 .

[13]  Davide Scaramuzza,et al.  EMVS: Event-Based Multi-View Stereo—3D Reconstruction with an Event Camera in Real-Time , 2017, International Journal of Computer Vision.

[14]  Melvyn L. Smith,et al.  Automated monitoring of dairy cow body condition, mobility and weight using a single 3D video capture device , 2018, Comput. Ind..

[15]  S. Foix,et al.  Lock-in Time-of-Flight (ToF) Cameras: A Survey , 2011, IEEE Sensors Journal.

[16]  Michael S. Brown,et al.  High quality depth map upsampling for 3D-TOF cameras , 2011, 2011 International Conference on Computer Vision.

[17]  Sebastian Thrun,et al.  An Application of Markov Random Fields to Range Sensing , 2005, NIPS.

[18]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Michelangelo Ceci,et al.  Semi-Supervised Multi-View Learning for Gene Network Reconstruction , 2015, SEBD.

[20]  Yi Chen,et al.  Wasserstein blue noise sampling , 2017, TOGS.

[21]  M. Gardiner,et al.  3D bioprinting for reconstructive surgery: Principles, applications and challenges. , 2017, Journal of plastic, reconstructive & aesthetic surgery : JPRAS.

[22]  Pengjiang Qian,et al.  SSC-EKE: Semi-supervised classification with extensive knowledge exploitation , 2018, Inf. Sci..

[23]  Lu Fang,et al.  SurfaceNet: An End-to-End 3D Neural Network for Multiview Stereopsis , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[24]  Pengjiang Qian,et al.  Recognition of Epileptic EEG Signals Using a Novel Multiview TSK Fuzzy System , 2017, IEEE Transactions on Fuzzy Systems.

[25]  Minh N. Do,et al.  A revisit to MRF-based depth map super-resolution and enhancement , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[26]  Jaime Laviada,et al.  Multiview three-dimensional reconstruction by millimetre-wave portable camera , 2017, Scientific Reports.

[27]  Ersin Yumer,et al.  3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[28]  Ting Luo,et al.  Blind quality estimator for 3D images based on binocular combination and extreme learning machine , 2017, Pattern Recognit..

[29]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[30]  Mark E. Campbell,et al.  Segmentation of dense range information in complex urban scenes , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[31]  Horst Bischof,et al.  OctNetFusion: Learning Depth Fusion from Data , 2017, 2017 International Conference on 3D Vision (3DV).

[32]  Achintya K. Bhowmik,et al.  3D computer vision based on machine learning with deep neural networks: A review , 2017 .

[33]  Pengjiang Qian,et al.  Multi-View Maximum Entropy Clustering by Jointly Leveraging Inter-View Collaborations and Intra-View-Weighted Attributes , 2018, IEEE Access.

[34]  Navneet Kumar,et al.  3D freehand ultrasound without external tracking using deep learning , 2018, Medical Image Anal..

[35]  C. Deutsch,et al.  Teacher's Aide Variogram Interpretation and Modeling , 2001 .

[36]  Gang Chen,et al.  Robust, Efficient Depth Reconstruction With Hierarchical Confidence-Based Matching , 2017, IEEE Transactions on Image Processing.

[37]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  Pengjiang Qian,et al.  Cross-domain, soft-partition clustering with diversity measure and knowledge reference , 2016, Pattern Recognit..

[39]  Yun Liang,et al.  Learning 3D faces from 2D images via Stacked Contractive Autoencoder , 2017, Neurocomputing.

[40]  J Frank,et al.  Three-dimensional reconstruction of single particles embedded in ice. , 1992, Ultramicroscopy.

[41]  Gernot Riegler,et al.  OctNet: Learning Deep 3D Representations at High Resolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  P. Gilbert.  Iterative methods for the three-dimensional reconstruction of an object from projections. , 1972, Journal of theoretical biology.

[43]  Sebastian Thrun,et al.  A Noise‐aware Filter for Real‐time Depth Upsampling , 2008 .

[44]  Zhaohong Deng,et al.  Multitask TSK Fuzzy System Modeling by Mining Intertask Common Hidden Structure , 2015, IEEE Transactions on Cybernetics.

[45]  Yizhou Yu,et al.  DeepSketch2Face , 2017, ACM Trans. Graph..

[46]  Raymond F. Muzic,et al.  Knowledge-leveraged transfer fuzzy C-Means for texture image segmentation with self-adaptive cluster prototype matching , 2017, Knowl. Based Syst..

[47]  Edmond Boyer,et al.  Shape Reconstruction Using Volume Sweeping and Learned Photoconsistency , 2018, ECCV.

[48]  Jun Yue,et al.  Supervised multiview learning based on simultaneous learning of multiview intact and single view classifier , 2016, Neural Computing and Applications.

[49]  Pengjiang Qian,et al.  Affinity and Penalty Jointly Constrained Spectral Clustering With All-Compatibility, Flexibility, and Robustness , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[50]  Changde Du,et al.  Reconstructing Perceived Images From Human Brain Activities With Bayesian Deep Multiview Learning , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[51]  Patrick Pérez,et al.  State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications , 2018, Comput. Graph. Forum.

[52]  Qiang Li,et al.  Accurate stereo matching based on weighted nonlocal aggregation for enhanced disparity refinement , 2018 .

[53]  Glen Berseth,et al.  DeepLoco , 2017, ACM Trans. Graph..

[54]  Pengjiang Qian,et al.  Fast Graph-Based Relaxed Clustering for Large Data Sets Using Minimal Enclosing Ball , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[55]  Peter Eisert,et al.  Multi‐view reconstruction of dynamic real‐world objects and their integration in augmented and virtual reality applications , 2017 .

[56]  Andrew Zisserman,et al.  SilNet : Single- and Multi-View Reconstruction by Learning from Silhouettes , 2017, BMVC.

[57]  Long Quan,et al.  MVSNet: Depth Inference for Unstructured Multi-view Stereo , 2018, ECCV.

[58]  Zhiguo Jiang,et al.  3D Reconstruction of Space Objects from Multi-Views by a Visible Sensor , 2017, Sensors.

[59]  Paul Newman,et al.  Image and Sparse Laser Fusion for Dense Scene Reconstruction , 2009, FSR.

[60]  Z Yin,et al.  An ab initio algorithm for low-resolution 3-D reconstructions from cryoelectron microscopy images. , 2001, Journal of structural biology.

[61]  Ruigang Yang,et al.  Spatial-Temporal Fusion for High Accuracy Depth Maps Using Dynamic MRFs , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[62]  Pengjiang Qian,et al.  Collaborative Fuzzy Clustering From Multiple Weighted Views , 2015, IEEE Transactions on Cybernetics.