TreePartNet

We present TreePartNet, a neural network aimed at reconstructing tree geometry from point clouds obtained by scanning real trees. Our key idea is to learn a natural neural decomposition exploiting the assumption that a tree comprises locally cylindrical shapes. In particular, reconstruction is a two-step process. First, two networks are used to detect priors from the point clouds. One detects semantic branching points, and the other network is trained to learn a cylindrical representation of the branches. In the second step, we apply a neural merging module to reduce the cylindrical representation to a final set of generalized cylinders combined by branches. We demonstrate results of reconstructing realistic tree geometry for a variety of input models and with varying input point quality, e.g., noise, outliers, and incompleteness. We evaluate our approach extensively by using data from both synthetic and real trees and comparing it with alternative methods.

[1]  D. Michels,et al.  Learning to reconstruct botanical trees from single images , 2021, ACM Trans. Graph..

[2]  Jianwei Guo,et al.  Single Image Tree Reconstruction via Adversarial Network , 2021, Graph. Model..

[3]  Shimin Hu,et al.  PCT: Point cloud transformer , 2020, Computational Visual Media.

[4]  Klaus Dietmayer,et al.  Point Transformer , 2020, IEEE Access.

[5]  N. Mitra,et al.  Points2Surf Learning Implicit Surfaces from Point Clouds , 2020, ECCV.

[6]  Bedrich Benes,et al.  Inverse Procedural Modeling of Branching Structures by Inferring L-Systems , 2020, ACM Trans. Graph..

[7]  Daniel Cohen-Or,et al.  Point2Mesh , 2020, ACM Trans. Graph..

[8]  E. Kalogerakis,et al.  RigNet , 2020, ACM Trans. Graph..

[9]  Raja Giryes,et al.  PointGMM: A Neural GMM Network for Point Clouds , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  E. Learned-Miller,et al.  Label-Efficient Learning on Point Clouds using Approximate Convex Decompositions , 2020, ECCV.

[11]  Zhuguo Li,et al.  PointASNL: Robust Point Clouds Processing Using Nonlocal Neural Networks With Adaptive Sampling , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Xiaopeng Zhang,et al.  Realistic Procedural Plant Modeling from Multiple View Images , 2020, IEEE Transactions on Visualization and Computer Graphics.

[13]  Zygmunt Pizlo,et al.  Skeleton Extraction from 3D Point Clouds by Decomposing the Object into Parts , 2019, ArXiv.

[14]  Thomas Funkhouser,et al.  Local Deep Implicit Functions for 3D Shape , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Y. Lipman,et al.  SAL: Sign Agnostic Learning of Shapes From Raw Data , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Hao Zhang,et al.  BSP-Net: Generating Compact Meshes via Binary Space Partitioning , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Geoffrey E. Hinton,et al.  CvxNet: Learnable Convex Decomposition , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Hugo Ledoux,et al.  AdTree: Accurate, Detailed, and Automatic Modelling of Laser-Scanned Trees , 2019, Remote. Sens..

[19]  Thomas A. Funkhouser,et al.  Learning Shape Templates With Structured Implicit Functions , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Duc Thanh Nguyen,et al.  JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds With Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Xiaoguang Han,et al.  A Skeleton-Bridged Deep Learning Approach for Generating Meshes of Complex Topologies From Single RGB Images , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Richard A. Newcombe,et al.  DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Sebastian Nowozin,et al.  Occupancy Networks: Learning 3D Reconstruction in Function Space , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Hao Zhang,et al.  Learning Implicit Fields for Generative Shape Modeling , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Gordon Wetzstein,et al.  DeepVoxels: Learning Persistent 3D Feature Embeddings , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Fuxin Li,et al.  PointConv: Deep Convolutional Networks on 3D Point Clouds , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Tamy Boubekeur,et al.  A Survey of Simple Geometric Primitives Detection Methods for Captured 3D Data , 2018, Comput. Graph. Forum.

[28]  Timo Ropinski,et al.  Monte Carlo convolution for learning on non-uniformly sampled point clouds , 2018, ACM Trans. Graph..

[29]  Yasushi Yagi,et al.  Probabilistic Plant Modeling via Multi-view Image-to-Image Translation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Wei Liu,et al.  Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images , 2018, ECCV.

[31]  Yaron Lipman,et al.  Point convolutional neural networks by extension operators , 2018, ACM Trans. Graph..

[32]  Daniel Cohen-Or,et al.  P2P-NET , 2018, ACM Trans. Graph..

[33]  Jitendra Malik,et al.  Learning Category-Specific Mesh Reconstruction from Image Collections , 2018, ECCV.

[34]  Xiaopeng Zhang,et al.  Tree Growth Modelling Constrained by Growth Equations , 2018, Comput. Graph. Forum.

[35]  Yue Wang,et al.  Dynamic Graph CNN for Learning on Point Clouds , 2018, ACM Trans. Graph..

[36]  Daniel Cremers,et al.  Clustering with Deep Learning: Taxonomy and New Methods , 2018, ArXiv.

[37]  Wei Wu,et al.  PointCNN: Convolution On X-Transformed Points , 2018, NeurIPS.

[38]  Ulrich Neumann,et al.  SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[39]  Maks Ovsjanikov,et al.  PCPNet Learning Local Shape Properties from Raw Point Clouds , 2017, Comput. Graph. Forum.

[40]  Kaiming He,et al.  Focal Loss for Dense Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[41]  Tamy Boubekeur,et al.  Bounding proxies for shape approximation , 2017, ACM Trans. Graph..

[42]  Subhransu Maji,et al.  3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks , 2017, 2017 International Conference on 3D Vision (3DV).

[43]  Leonidas J. Guibas,et al.  Learning Representations and Generative Models for 3D Point Clouds , 2017, ICML.

[44]  S. Lucey,et al.  Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction , 2017, AAAI.

[45]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[46]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[47]  Thomas Brox,et al.  Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[48]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Hao Su,et al.  A Point Set Generation Network for 3D Object Reconstruction from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Leonidas J. Guibas,et al.  Learning Shape Abstractions by Assembling Volumetric Primitives , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Carlos Andújar,et al.  Single-picture reconstruction and rendering of trees for plausible vegetation synthesis , 2016, Comput. Graph..

[52]  Ali Farhadi,et al.  Unsupervised Deep Embedding for Clustering Analysis , 2015, ICML.

[53]  Daniel Cohen-Or,et al.  Generalized cylinder decomposition , 2015, ACM Trans. Graph..

[54]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[55]  Long Quan,et al.  Data-Driven Synthetic Modeling of Trees , 2014, IEEE Transactions on Visualization and Computer Graphics.

[56]  Radomír Mech,et al.  Inverse Procedural Modelling of Trees , 2014, Comput. Graph. Forum.

[57]  Alla Sheffer,et al.  PolyCut , 2013, ACM Trans. Graph..

[58]  Daniel Cohen-Or,et al.  L1-medial skeleton of point cloud , 2013, ACM Trans. Graph..

[59]  Paul A. Beardsley,et al.  Image-based reconstruction and synthesis of dense foliage , 2013, ACM Trans. Graph..

[60]  Michael M. Kazhdan,et al.  Screened poisson surface reconstruction , 2013, TOGS.

[61]  Min Liu,et al.  Towards locally and globally shape-aware reverse 3D modeling , 2012, Comput. Aided Des..

[62]  P. Hall,et al.  Modeling and generating moving trees from video , 2011, ACM Trans. Graph..

[63]  Daniel Cohen-Or,et al.  Texture-lobes for tree modelling , 2011, ACM Trans. Graph..

[64]  Matt Olson,et al.  Automatic reconstruction of tree skeletal structures from point clouds , 2010, ACM Trans. Graph..

[65]  Daniel G. Aliaga,et al.  Inverse Procedural Modeling by Automatic Generation of L‐systems , 2010, Comput. Graph. Forum.

[66]  Dong-Ming Yan,et al.  Efficient and robust reconstruction of botanical branching structure from laser scanned points , 2009, 2009 11th IEEE International Conference on Computer-Aided Design and Computer Graphics.

[67]  Radomír Mech,et al.  Self-organizing tree models for image synthesis , 2009, ACM Trans. Graph..

[68]  Bedrich Benes,et al.  Interactive Modeling of Virtual Ecosystems , 2009, NPH.

[69]  Long Quan,et al.  Single image tree modeling , 2008, ACM Trans. Graph..

[70]  Hui Xu,et al.  Knowledge and heuristic-based modeling of laser-scanned trees , 2007, TOGS.

[71]  Adam Runions,et al.  Modeling Trees with a Space Colonization Algorithm , 2007, NPH.

[72]  Oliver Deussen,et al.  Approximate image-based tree-modeling using particle flows , 2007, ACM Trans. Graph..

[73]  S. B. Kang,et al.  Image-based tree modeling , 2007, ACM Trans. Graph..

[74]  Long Quan,et al.  Image-based plant modeling , 2006, ACM Trans. Graph..

[75]  Michael M. Kazhdan,et al.  Poisson surface reconstruction , 2006, SGP '06.

[76]  George Drettakis,et al.  Volumetric reconstruction and interactive rendering of trees from photographs , 2004, ACM Trans. Graph..

[77]  Julie Dorsey,et al.  Reconstructing 3D Tree Models from Instrumented Photographs , 2001, IEEE Computer Graphics and Applications.

[78]  Tiow Seng Tan,et al.  Decomposing polygon meshes for interactive applications , 2001, I3D '01.

[79]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[80]  James H. Brown,et al.  A general model for the structure and allometry of plant vascular systems , 1999, Nature.

[81]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[82]  Hans-Peter Kriegel,et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise , 1996, KDD.

[83]  Marc Jaeger,et al.  Plant models faithful to botanical structure and development , 1988, SIGGRAPH.

[84]  Przemyslaw Prusinkiewicz,et al.  Graphical applications of L-systems , 1986 .

[85]  H. Honda Description of the form of trees by the parameters of the tree-like body: effects of the branching angle and the branch length on the sample of the tree-like body. , 1971, Journal of theoretical biology.

[86]  A. Lindenmayer Mathematical models for cellular interactions in development. I. Filaments with one-sided inputs. , 1968, Journal of theoretical biology.

[87]  Logan LOGAN: Unpaired Shape Transform in Latent Overcomplete Space , 2019 .

[88]  A. Lindenmayer,et al.  The Algorithmic Beauty of Plants , 1990, The Virtual Laboratory.

[89]  Larry D. Hostetler,et al.  The estimation of the gradient of a density function, with applications in pattern recognition , 1975, IEEE Trans. Inf. Theory.

[90]  Aristid Lindenmayer,et al.  Mathematical Models for Cellular Interactions in Development , 1968 .

[91]  Kangxue Yin,et al.  P2P-NET: Bidirectional Point Displacement Net for Shape Transform , 2022 .

[92]  Hao Zhang,et al.  Ieee International Conference on Shape Modeling and Applications (smi) 2010 Point Cloud Skeletons via Laplacian-based Contraction , 2022 .