Pixel2Mesh: 3D Mesh Model Generation via Image Guided Deformation

In this paper, we propose an end-to-end deep learning architecture that generates 3D triangular meshes from single color images. Limited by the nature of the prevalent deep learning techniques, the majority of previous works usually represent 3D shapes in 3D volumes or point clouds. However, it is non-trivial to convert them to compact and ready-to-use mesh models. Unlike the existing methods, our network represents 3D shapes in meshes, which are essentially graphs and well suited for graph-based convolutional neural network. Leveraging on perceptual features extracted from the input image, our network produces correct geometry by progressively deforming an ellipsoid. To make the whole deformation procedure stable, we adopt a coarse-to-fine strategy, and define various mesh/surface related losses to capture properties of different levels; this guarantees visually appealing and physically accurate 3D geometry. In addition to producing accurate 3D shape on the 3D ShapeNet dataset, our model by nature can be adapted to objects in specific domains, e.g. human face, and easily extended to learn per-vertex properties, e.g. color. Extensive experiments show that our method not only qualitatively produces mesh model with better details, but also achieves higher 3D shape estimation accuracy compared to the state-of-the-art.