Go with the Flows: Mixtures of Normalizing Flows for Point Cloud Generation and Reconstruction

Recently Normalizing Flows (NFs) have demonstrated state-of-the-art performance on modeling 3D point clouds while allowing sampling with arbitrary resolution at inference time. However, these flow-based models still have fundamental limitations on complicated geometries. This work generalizes prior work by introducing additional discrete latent variable, i.e. mixture model. This circumvents limitations of prior approaches, leads to more parameter efficient models and reduces the inference runtime. Moreover, in this more general framework each component learns to specialize in a particular subregion of an object in a completely unsupervised fashion yielding promising clustering properties. We further demonstrate that by adding data augmentation, individual mixture components can learn to specialize in a semantically meaningful manner. We evaluate mixtures of NFs on generation, autoencoding and single-view reconstruction based on the ShapeNet dataset.

[1]  Mohammed Bennamoun,et al.  Deep Learning for 3D Point Clouds: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.

[3]  Ming-Yu Liu,et al.  PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[4]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[5]  Donald D. Hoffman,et al.  Parts of recognition , 1984, Cognition.

[6]  Yaron Lipman,et al.  Multi-chart generative surface modeling , 2018, ACM Trans. Graph..

[7]  Janis Postels,et al.  Variational Transformer Networks for Layout Generation , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[9]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Piotr Klukowski,et al.  Adversarial autoencoders for compact representations of 3D point clouds , 2018, Comput. Vis. Image Underst..

[11]  K. Cranmer,et al.  Flows for simultaneous manifold learning and density estimation , 2020, Neural Information Processing Systems.

[12]  Enrico Magli,et al.  Learning Localized Generative Models for 3D Point Clouds via Graph Convolution , 2018, ICLR.

[13]  V. Ferrari,et al.  C-Flow: Conditional Generative Flow Models for Images and 3D Point Clouds , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Wei Liu,et al.  Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images , 2018, ECCV.

[15]  Ullrich Köthe,et al.  Guided Image Generation with Conditional Invertible Neural Networks , 2019, ArXiv.

[16]  Joun Yeop Lee,et al.  SoftFlow: Probabilistic Framework for Normalizing Flow on Manifolds , 2020, NeurIPS.

[17]  Hao Su,et al.  A Point Set Generation Network for 3D Object Reconstruction from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[19]  L. Gool,et al.  SRFlow: Learning the Super-Resolution Space with Normalizing Flow , 2020, ECCV.

[20]  Razvan Pascanu,et al.  A RAD approach to deep mixture models , 2019, DGS@ICLR.

[21]  Jacek Tabor,et al.  HyperFlow: Representing 3D Objects as Surfaces , 2020, ArXiv.

[22]  Subhransu Maji,et al.  Multiresolution Tree Networks for 3D Point Cloud Processing , 2018, ECCV.

[23]  Luc Van Gool,et al.  DeFlow: Learning Complex Image Degradations from Unpaired Data with Conditional Flows , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  S. Fidler,et al.  Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Dong Tian,et al.  FoldingNet: Point Cloud Auto-Encoder via Deep Grid Deformation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Pieter Abbeel,et al.  Variational Lossy Autoencoder , 2016, ICLR.

[27]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[28]  Max Welling,et al.  Improved Variational Inference with Inverse Autoregressive Flow , 2016, NIPS 2016.

[29]  Andrew Gordon Wilson,et al.  Semi-Supervised Learning with Normalizing Flows , 2019, ICML.

[30]  Aaron C. Courville,et al.  FiLM: Visual Reasoning with a General Conditioning Layer , 2017, AAAI.

[31]  Anthony L. Caterini,et al.  Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows , 2019, ICML.

[32]  Mathieu Aubry,et al.  A Papier-Mache Approach to Learning 3D Surface Generation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[33]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  David Duvenaud,et al.  FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models , 2018, ICLR.

[35]  Robert Giaquinto,et al.  Gradient Boosted Normalizing Flows , 2020, NeurIPS.

[36]  Yue Wang,et al.  PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attention , 2018, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[37]  Leonidas J. Guibas,et al.  ShapeNet: An Information-Rich 3D Model Repository , 2015, ArXiv.

[38]  Silvio Savarese,et al.  3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction , 2016, ECCV.

[39]  Thomas Brox,et al.  What Do Single-View 3D Reconstruction Networks Learn? , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Leonidas J. Guibas,et al.  Learning Representations and Generative Models for 3D Point Clouds , 2017, ICML.

[41]  Edmond Boyer,et al.  Discrete Point Flow Networks for Efficient Point Cloud Generation , 2020, ECCV.

[42]  J. Knott.  The organization of behavior: A neuropsychological theory , 1951 .

[43]  Seong Joon Oh,et al.  Reliable Fidelity and Diversity Metrics for Generative Models , 2020, ICML.

[44]  Shitong Luo,et al.  Diffusion Probabilistic Models for 3D Point Cloud Generation , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Kui Jia,et al.  Deep Cascade Generation on Point Sets , 2019, IJCAI.

[46]  Junseok Kwon,et al.  3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[47]  Takashi Matsubara,et al.  ChartPointFlow for Topology-Aware 3D Point Cloud Generation , 2021, ACM Multimedia.

[48]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Yoshua Bengio,et al.  NICE: Non-linear Independent Components Estimation , 2014, ICLR.

[50]  Yue Wang,et al.  Dynamic Graph CNN for Learning on Point Clouds , 2018, ACM Trans. Graph..

[51]  Noah Snavely,et al.  Learning Gradient Fields for Shape Generation , 2020, ECCV.

[52]  Jan Chorowski,et al.  Conditional Invertible Flow for Point Cloud Generation , 2019, ArXiv.

[53]  Mário A. T. Figueiredo,et al.  Variational MIxture of Normalizing Flows , 2020, ESANN.

[54]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.