Graph Signal Processing for Geometric Data and Beyond: Theory and Applications

Geometric data acquired from real-world scenes, e.g., 2D depth images, 3D point clouds, and 4D dynamic point clouds, have found a wide range of applications including immersive telepresence, autonomous driving, surveillance, etc. Due to irregular sampling patterns of most geometric data, traditional image/video processing methodologies are limited, while Graph Signal Processing (GSP)---a fast-developing field in the signal processing community---enables processing signals that reside on irregular domains and plays a critical role in numerous applications of geometric data from low-level processing to high-level analysis. To further advance the research in this field, we provide the first timely and comprehensive overview of GSP methodologies for geometric data in a unified manner by bridging the connections between geometric data and graphs, among the various geometric data modalities, and with spectral/nodal graph filtering techniques. We also discuss the recently developed Graph Neural Networks (GNNs) and interpret the operation of these networks from the perspective of GSP. We conclude with a brief discussion of open problems and challenges.

[1]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[2]  Yu-Chiang Frank Wang,et al.  Convolution in the Cloud: Learning Deformable Kernels in 3D Graph Convolution Networks for Point Cloud Analysis , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Yifan Zhang,et al.  Skeleton-Based Action Recognition With Multi-Stream Adaptive Graph Convolutional Networks , 2019, IEEE Transactions on Image Processing.

[4]  Stephen P. Boyd,et al.  An ADMM Algorithm for a Class of Total Variation Regularized Estimation Problems , 2012, 1203.1828.

[5]  Lei Wang,et al.  Appendix for : Graph Attention Convolution for Point Cloud Semantic Segmentation , 2019 .

[6]  Dong Tian,et al.  Attribute compression for sparse point clouds using graph transforms , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[7]  Antonio Ortega,et al.  Edge-adaptive depth map coding with lifting transform on graphs , 2015, 2015 Picture Coding Symposium (PCS).

[8]  Philippe Coiffet,et al.  Virtual Reality Technology , 2003, Presence: Teleoperators & Virtual Environments.

[9]  Oscar C. Au,et al.  Depth map compression using multi-resolution graph-based transform for depth-image-based rendering , 2012, 2012 19th IEEE International Conference on Image Processing.

[10]  Wei Hu,et al.  Dynamic Point Cloud Inpainting via Spatial-Temporal Graph Learning , 2021, IEEE Transactions on Multimedia.

[11]  Ling Huang,et al.  An Analysis of the Convergence of Graph Laplacians , 2010, ICML.

[12]  Wei Hu,et al.  GraphTER: Unsupervised Learning of Graph Transformation Equivariant Representations via Auto-Encoding Node-Wise Transformations , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Wei Hu,et al.  Exploring Structure-Adaptive Graph Learning for Robust Semi-Supervised Classification , 2019, 2020 IEEE International Conference on Multimedia and Expo (ICME).

[15]  Wei Hu,et al.  Local Frequency Interpretation and Non-Local Self-Similarity on Graph for Point Cloud Inpainting , 2018, IEEE Transactions on Image Processing.

[16]  Ming Hao,et al.  Linked Dynamic Graph CNN: Learning on Point Cloud via Linking Hierarchical Features , 2019, ArXiv.

[17]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[18]  Karen O. Egiazarian,et al.  Image denoising with block-matching and 3D filtering , 2006, Electronic Imaging.

[19]  Antonio Ortega,et al.  Intra-Prediction and Generalized Graph Fourier Transform for Image Coding , 2015, IEEE Signal Processing Letters.

[20]  Pierre Vandergheynst,et al.  Wavelets on Graphs via Spectral Graph Theory , 2009, ArXiv.

[21]  Joan Bruna,et al.  Deep Convolutional Networks on Graph-Structured Data , 2015, ArXiv.

[22]  Enrico Magli,et al.  Learning Localized Generative Models for 3D Point Clouds via Graph Convolution , 2018, ICLR.

[23]  Silvio Savarese,et al.  3D Semantic Parsing of Large-Scale Indoor Spaces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[25]  S. Helgason.  Differential Geometry, Lie Groups, and Symmetric Spaces , 1978 .

[26]  Junseok Kwon,et al.  3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[27]  Wen Gao,et al.  Predictive Generalized Graph Fourier Transform for Attribute Compression of Dynamic Point Clouds , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[28]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[29]  Santiago Segarra,et al.  Connecting the Dots: Identifying Network Structure via Graph Signal Processing , 2018, IEEE Signal Processing Magazine.

[30]  Siheng Chen,et al.  3D Point Cloud Processing and Learning for Autonomous Driving , 2020, ArXiv.

[31]  Sanja Fidler,et al.  3D Graph Neural Networks for RGBD Semantic Segmentation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[32]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[33]  Philip A. Chou,et al.  Transform Coding for Point Clouds Using a Gaussian Process Model , 2017, IEEE Transactions on Image Processing.

[34]  Pascal Frossard,et al.  The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains , 2012, IEEE Signal Processing Magazine.

[35]  Reinhard Klein,et al.  Eurographics Symposium on Point-based Graphics (2006) Octree-based Point-cloud Compression , 2022 .

[36]  Nikos Komodakis,et al.  Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Wei Hu,et al.  RGLN: Robust Residual Graph Learning Networks via Similarity-Preserving Mapping on Graphs , 2021, ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[38]  Vincent Gripon,et al.  An Inside Look at Deep Neural Networks Using Graph Signal Processing , 2018, 2018 Information Theory and Applications Workshop (ITA).

[39]  Pierre Vandergheynst,et al.  Graph-based denoising for time-varying point clouds , 2015, 2015 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[40]  Kilian Q. Weinberger,et al.  Simplifying Graph Convolutional Networks , 2019, ICML.

[41]  O. Axelsson,et al.  On the rate of convergence of the preconditioned conjugate gradient method , 1986 .

[42]  Dong Tian,et al.  Mining Point Cloud Local Structures by Kernel Correlation and Graph Pooling , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[43]  Thomas Maugey,et al.  Graph-Based Representation for Multiview Image Geometry , 2015, IEEE Transactions on Image Processing.

[44]  Csaba Benedek,et al.  3D people surveillance on range data sequences of a rotating Lidar , 2014, Pattern Recognit. Lett..

[45]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[46]  Ling Zhang,et al.  Unsupervised Feature Learning for Point Cloud Understanding by Contrasting and Clustering Using Graph Convolutional Neural Networks , 2019, 2019 International Conference on 3D Vision (3DV).

[47]  Kaveh Hassani,et al.  Unsupervised Multi-Task Feature Learning on Point Clouds , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[48]  Pierre Vandergheynst,et al.  Geometric Deep Learning: Going beyond Euclidean data , 2016, IEEE Signal Process. Mag..

[49]  Xianming Liu,et al.  Random Walk Graph Laplacian-Based Smoothness Prior for Soft Decoding of JPEG Images , 2016, IEEE Transactions on Image Processing.

[50]  Derek Hoiem,et al.  Indoor Segmentation and Support Inference from RGBD Images , 2012, ECCV.

[51]  Xiaowen Dong,et al.  Graph Signal Processing for Machine Learning: A Review and New Perspectives , 2020, IEEE Signal Processing Magazine.

[52]  Dong Tian,et al.  Deep Unsupervised Learning of 3D Point Clouds via Graph Topology Inference and Filtering , 2019, IEEE Transactions on Image Processing.

[53]  José M. F. Moura,et al.  Spectral Projector-Based Graph Fourier Transforms , 2017, IEEE Journal of Selected Topics in Signal Processing.

[54]  Subhransu Maji,et al.  Multi-view Convolutional Neural Networks for 3D Shape Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[55]  Wen Gao,et al.  Graph-Based Blind Image Deblurring From a Single Photograph , 2018, IEEE Transactions on Image Processing.

[56]  Andrew Knyazev,et al.  Chebyshev and conjugate gradient filters for graph image denoising , 2014, 2014 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[57]  Pierre Vandergheynst,et al.  Graph Signal Processing: Overview, Challenges, and Applications , 2017, Proceedings of the IEEE.

[58]  Dong Tian,et al.  FoldingNet: Point Cloud Auto-Encoder via Deep Grid Deformation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[59]  Fan Chung,et al.  Spectral Graph Theory , 1996 .

[60]  Antonio Ortega,et al.  A graph-based joint bilateral approach for depth enhancement , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[61]  Oscar C. Au,et al.  Multiresolution Graph Fourier Transform for Compression of Piecewise Smooth Images , 2015, IEEE Transactions on Image Processing.

[62]  Nicolas Tremblay,et al.  Approximate Fast Graph Fourier Transforms via Multilayer Sparse Approximations , 2016, IEEE Transactions on Signal and Information Processing over Networks.

[63]  Zhu Li,et al.  Attribute compression of 3D point clouds using Laplacian sparsity optimized graph transform , 2017, 2017 IEEE Visual Communications and Image Processing (VCIP).

[64]  Pascal Frossard,et al.  Learning Graphs From Data: A Signal Representation Perspective , 2018, IEEE Signal Processing Magazine.

[65]  Matthias Nießner,et al.  ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[66]  A. Singer.  From graph to manifold Laplacian: The convergence rate , 2006 .

[67]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[68]  Jiaxin Li,et al.  SO-Net: Self-Organizing Network for Point Cloud Analysis , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[69]  Antonio Ortega,et al.  Depth map coding using graph based transform and transform domain sparsification , 2011, 2011 IEEE 13th International Workshop on Multimedia Signal Processing.

[70]  Amin Zheng,et al.  RGCNN: Regularized Graph CNN for Point Cloud Segmentation , 2018, ACM Multimedia.

[71]  Kaleem Siddiqi,et al.  Local Spectral Graph Convolution for Point Set Feature Learning , 2018, ECCV.

[72]  Cyrill Stachniss,et al.  SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[73]  Wei An,et al.  Learning Multi-View Representation With LSTM for 3-D Shape Recognition and Retrieval , 2019, IEEE Transactions on Multimedia.

[74]  Touradj Ebrahimi,et al.  JPEG Pleno: Toward an Efficient Representation of Visual Reality , 2016, IEEE MultiMedia.

[75]  Matthias Hein,et al.  Manifold Denoising , 2006, NIPS.

[76]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[77]  Gene Cheung,et al.  Point Cloud Denoising via Feature Graph Laplacian Regularization , 2020, IEEE Transactions on Image Processing.

[78]  Gabriel Taubin,et al.  A benchmark for surface reconstruction , 2013, TOGS.

[79]  Ljubisa Stankovic,et al.  Graph Signal Processing - Part III: Machine Learning on Graphs, from Graph Topology to Applications , 2020, ArXiv.

[80]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[81]  Ulrike von Luxburg,et al.  Graph Laplacians and their Convergence on Random Neighborhood Graphs , 2006, J. Mach. Learn. Res..

[82]  Wei Wu,et al.  PointCNN: Convolution On X-Transformed Points , 2018, NeurIPS.

[83]  Mohammed Bennamoun,et al.  Deep Learning for 3D Point Clouds: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[84]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[85]  Dacheng Tao,et al.  Context Aware Graph Convolution for Skeleton-Based Action Recognition , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[86]  Wei Hu,et al.  3d Dynamic Point Cloud Inpainting Via Temporal Consistency On Graphs , 2020, 2020 IEEE International Conference on Multimedia and Expo (ICME).

[87]  Charles T. Loop,et al.  Point cloud attribute compression with graph transform , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[88]  Sunil K. Narang,et al.  Bilateral filter: Graph spectral interpretation and extensions , 2013, 2013 IEEE International Conference on Image Processing.

[89]  José M. F. Moura,et al.  Discrete Signal Processing on Graphs , 2012, IEEE Transactions on Signal Processing.

[90]  Suhang Wang,et al.  Graph Structure Learning for Robust Graph Neural Networks , 2020, KDD.

[91]  Jian Zhang,et al.  Understanding Graph Neural Networks from Graph Signal Denoising Perspectives , 2020, ArXiv.

[92]  Jaejoon Lee,et al.  Edge-adaptive transforms for efficient depth map coding , 2010, 28th Picture Coding Symposium.

[93]  Wei Hu,et al.  Differentiable Manifold Reconstruction for Point Cloud Denoising , 2020, ACM Multimedia.

[94]  Gene Cheung,et al.  Deep Graph Laplacian Regularization for Robust Denoising of Real Images , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[95]  Michael G. Rabbat,et al.  A Graph-CNN for 3D Point Cloud Classification , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[96]  Matthias Hein,et al.  Uniform Convergence of Adaptive Graph-Based Regularization , 2006, COLT.

[97]  Antonio Ortega,et al.  Graph Learning From Data Under Laplacian and Structural Constraints , 2016, IEEE Journal of Selected Topics in Signal Processing.

[98]  Philip A. Chou,et al.  Graph Signal Processing – A Probabilistic Framework , 2016 .

[99]  Joan Bruna,et al.  Spectral Networks and Locally Connected Networks on Graphs , 2013, ICLR.

[100]  Oscar C. Au,et al.  Depth map denoising using graph-based transform and group sparsity , 2013, 2013 IEEE 15th International Workshop on Multimedia Signal Processing (MMSP).

[101]  Pascal Frossard,et al.  Graph-Based Compression of Dynamic 3D Point Cloud Sequences , 2015, IEEE Transactions on Image Processing.

[102]  Shrikanth S. Narayanan,et al.  Irregularity-Aware Graph Fourier Transforms , 2018, IEEE Transactions on Signal Processing.

[103]  Cha Zhang,et al.  Analyzing the Optimality of Predictive Transform Coding Using Graph-Based Models , 2013, IEEE Signal Processing Letters.

[104]  Ron Kimmel,et al.  Patch‐Collaborative Spectral Point‐Cloud Denoising , 2013, Comput. Graph. Forum.

[105]  Dong Tian,et al.  Point Cloud Attribute Compression Using 3-D Intra Prediction and Shape-Adaptive Transforms , 2016, 2016 Data Compression Conference (DCC).

[106]  Fei Wu,et al.  Spatio-Temporal Graph Routing for Skeleton-Based Action Recognition , 2019, AAAI.

[107]  Yue Wang,et al.  Dynamic Graph CNN for Learning on Point Clouds , 2018, ACM Trans. Graph..

[108]  Richard A. Newcombe,et al.  DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[109]  Bin Luo,et al.  Semi-Supervised Learning With Graph Learning-Convolutional Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[110]  Jiaying Liu,et al.  Optimized Skeleton-based Action Recognition via Sparsified Graph Regression , 2018, ACM Multimedia.

[111]  Wei Liu,et al.  Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images , 2018, ECCV.

[112]  Gene Cheung,et al.  Graph Laplacian Regularization for Image Denoising: Analysis in the Continuous Domain , 2016, IEEE Transactions on Image Processing.

[113]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[114]  Camille Couprie,et al.  Dual constrained TV-based regularization , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[115]  Jose Dolz,et al.  Laplacian Regularized Few-Shot Learning , 2020, ICML.

[116]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[117]  Pascal Frossard,et al.  Geometry-Consistent Light Field Super-Resolution via Graph-Based Regularization , 2017, IEEE Transactions on Image Processing.

[118]  Jianxiong Xiao,et al.  3D ShapeNets: A deep representation for volumetric shapes , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[119]  Sunil K. Narang,et al.  Graph based transforms for depth video coding , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[120]  Wei Hu,et al.  Feature Graph Learning for 3D Point Cloud Denoising , 2019, IEEE Transactions on Signal Processing.

[121]  Abderrahim Elmoataz,et al.  Nonlocal Discrete Regularization on Weighted Graphs: A Framework for Image and Manifold Processing , 2008, IEEE Transactions on Image Processing.

[122]  Chen Feng,et al.  Fast Resampling of Three-Dimensional Point Clouds via Graphs , 2017, IEEE Transactions on Signal Processing.

[123]  Enrico Magli,et al.  Deep Graph-Convolutional Image Denoising , 2019, IEEE Transactions on Image Processing.

[124]  Jonathan Masci,et al.  Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[125]  Silvio Savarese,et al.  4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[126]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[127]  Leonidas J. Guibas,et al.  ShapeNet: An Information-Rich 3D Model Repository , 2015, ArXiv.

[128]  Gene Cheung,et al.  3D Point Cloud Denoising Using Graph Laplacian Regularization of a Low Dimensional Manifold Model , 2018, IEEE Transactions on Image Processing.

[129]  Weijing Shi,et al.  Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[130]  Antonio Ortega,et al.  Fast Graph Fourier Transforms Based on Graph Symmetry and Bipartition , 2019, IEEE Transactions on Signal Processing.

[131]  Massimiliano Pontil,et al.  Learning Discrete Structures for Graph Neural Networks , 2019, ICML.

[132]  Dahua Lin,et al.  Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition , 2018, AAAI.

[133]  Ruoyu Li,et al.  Adaptive Graph Convolutional Neural Networks , 2018, AAAI.

[134]  Catarina Brites,et al.  Graph-Based Static 3D Point Clouds Geometry Coding , 2019, IEEE Transactions on Multimedia.

[135]  Wen Gao,et al.  Cluster-Based Point Cloud Coding with Normal Weighted Graph Fourier Transform , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[136]  Stanley Osher,et al.  A Unified Primal-Dual Algorithm Framework Based on Bregman Iteration , 2010, J. Sci. Comput..

[137]  Stanley Osher,et al.  Low Dimensional Manifold Model for Image Processing , 2017, SIAM J. Imaging Sci..

[138]  Antonio Ortega,et al.  Compression of dynamic 3D point clouds using subdivisional meshes and graph wavelet transforms , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[139]  Ji Wan,et al.  Multi-view 3D Object Detection Network for Autonomous Driving , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).