Vector-valued Distance and Gyrocalculus on the Space of Symmetric Positive Definite Matrices

We propose the use of the vector-valued distance to compute distances and extract geometric information from the manifold of symmetric positive definite matrices (SPD), and develop gyrovector calculus, constructing analogs of vector space operations in this curved space. We implement these operations and showcase their versatility in the tasks of knowledge graph completion, item recommendation, and question answering. In experiments, the SPD models outperform their equivalents in Euclidean and hyperbolic space. The vector-valued distance allows us to visualize embeddings, showing that the models learn to disentangle representations of positive samples from negative ones.

[1]  Guillaume Bouchard,et al.  Complex Embeddings for Simple Link Prediction , 2016, ICML.

[2]  Cristian Sminchisescu,et al.  Matrix Backpropagation for Deep Networks with Structured Layers , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[3]  Nicholas Jing Yuan,et al.  Collaborative Knowledge Base Embedding for Recommender Systems , 2016, KDD.

[4]  Matthieu Cord,et al.  Exploring Complex Time-series Representations for Riemannian Machine Learning of Radar Data , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[5]  Michael Strube,et al.  Augmenting the User-Item Graph with Textual Similarity Models , 2021, ArXiv.

[6]  P. Eberlein.  Structure of manifolds of nonpositive curvature , 1985 .

[7]  Yunde Jia,et al.  Learning a Robust Representation via a Deep Network on Symmetric Positive Definite Manifolds , 2017, Pattern Recognit..

[8]  Vittorio Murino,et al.  Multi-class Classification on Riemannian Manifolds for Video Surveillance , 2010, ECCV.

[9]  Yi Tay,et al.  Deep Learning based Recommender System: A Survey and New Perspectives , 2018 .

[10]  Yi Yang,et al.  WikiQA: A Challenge Dataset for Open-Domain Question Answering , 2015, EMNLP.

[11]  Jianmo Ni,et al.  Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects , 2019, EMNLP.

[12]  Vittorio Murino,et al.  Log-Hilbert-Schmidt metric between positive definite operators on Hilbert spaces , 2014, NIPS.

[13]  Danqi Chen,et al.  Observed versus latent features for knowledge base and text inference , 2015, CVSC.

[14]  Peter Dolog,et al.  MindReader: Recommendation over Knowledge Graph Entities with Explicit User Ratings , 2020, CIKM.

[15]  Søren Hauberg,et al.  Geodesic exponential kernels: When curvature and linearity conflict , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Fatih Murat Porikli,et al.  Pedestrian Detection via Classification on Riemannian Manifolds , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Levent Tunçel,et al.  Optimization algorithms on matrix manifolds , 2009, Math. Comput..

[18]  Tong Zhang,et al.  Deep Manifold-to-Manifold Transforming Network , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[19]  M. Kapovich,et al.  Anosov subgroups: dynamical and geometric characterizations , 2017, 1703.01647.

[20]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[21]  Xing Xie,et al.  A Survey on Knowledge Graph-Based Recommender Systems , 2020, IEEE Transactions on Knowledge and Data Engineering.

[22]  Examples and Applications of Generalized Gyrovector Spaces , 2017 .

[23]  Chi Zhang,et al.  Deep Manifold Learning of Symmetric Positive Definite Matrices with Application to Face Recognition , 2017, AAAI.

[24]  Michael Strube,et al.  A Fully Hyperbolic Neural Model for Hierarchical Multi-class Classification , 2020, FINDINGS.

[25]  Shiguang Shan,et al.  Log-Euclidean Metric Learning on Symmetric Positive Definite Manifold with Application to Image Set Classification , 2015, ICML.

[26]  Octavian-Eugen Ganea,et al.  Constant Curvature Graph Convolutional Networks , 2019, ICML.

[27]  Convex functions on symmetric spaces, side lengths of polygons and the stability inequalities for weighted configurations at infinity , 2003, math/0311486.

[28]  Timothy M. Hospedales,et al.  Multi-relational Poincaré Graph Embeddings , 2019, NeurIPS.

[29]  Gary Bécigneul,et al.  Riemannian Adaptive Optimization Methods , 2018, ICLR.

[30]  Rudrasis Chakraborty,et al.  ManifoldNet: A Deep Neural Network for Manifold-Valued Data With Applications , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Suvrit Sra,et al.  A new metric on the manifold of kernel matrices with application to matrix geometric means , 2012, NIPS.

[32]  Mehrtash Tafazzoli Harandi,et al.  From Manifold to Manifold: Geometry-Aware Dimensionality Reduction for SPD Matrices , 2014, ECCV.

[33]  Frank Hutter,et al.  Decoupled Weight Decay Regularization , 2017, ICLR.

[34]  Sejong Kim.  Gyrovector Spaces on the Open Convex Cone of Positive Definite Matrices , 2016 .

[35]  Xu Chen,et al.  Learning over Knowledge-Base Embeddings for Recommendation , 2018, Algorithms.

[36]  Douwe Kiela,et al.  Poincaré Embeddings for Learning Hierarchical Representations , 2017, NIPS.

[37]  Xu Chen,et al.  Learning over Knowledge-Base Embeddings for Recommendation , 2018, ArXiv.

[38]  Jian-Yun Nie,et al.  RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space , 2018, ICLR.

[39]  Siu Cheung Hui,et al.  Hyperbolic Representation Learning for Fast and Efficient Neural Question Answering , 2017, WSDM.

[40]  Yunde Jia,et al.  Manifold Kernel Sparse Representation of Symmetric Positive-Definite Matrices and Its Applications , 2015, IEEE Transactions on Image Processing.

[41]  Michael Strube,et al.  Symmetric Spaces for Graph Embeddings: A Finsler-Riemannian Approach , 2021, ICML.

[42]  Anqi Li,et al.  Generalized Nonlinear and Finsler Geometry for Robotics , 2020, IEEE International Conference on Robotics and Automation.

[43]  R. Bhatia.  On the exponential metric increasing property , 2003 .

[44]  Frank Nielsen,et al.  Clustering in Hilbert’s Projective Geometry: The Case Studies of the Probability Simplex and the Elliptope of Correlation Matrices , 2018, Geometric Structures of Information.

[45]  Abraham Albert Ungar,et al.  A Gyrovector Space Approach to Hyperbolic Geometry , 2009, A Gyrovector Space Approach to Hyperbolic Geometry.

[46]  R. Bhatia,et al.  On the Bures–Wasserstein distance between positive definite matrices , 2017, Expositiones Mathematicae.

[47]  Tong Yang,et al.  NagE: Non-Abelian Group Embedding for Knowledge Graphs , 2020, CIKM.

[48]  Xavier Pennec,et al.  A Riemannian Framework for Tensor Computing , 2005, International Journal of Computer Vision.

[49]  Fan Meng,et al.  Recommendation Algorithm based on Link Prediction and Domain Knowledge in Retail Transactions , 2014, ITQM.

[50]  S. Helgason.  Differential Geometry, Lie Groups, and Symmetric Spaces , 1978 .

[51]  Yangyang Li,et al.  Locality preserving projection on SPD matrix Lie group: algorithm and analysis , 2017, Science China Information Sciences.

[52]  Nicholas Ayache,et al.  Geometric Means in a Novel Vector Space Structure on Symmetric Positive-Definite Matrices , 2007, SIAM J. Matrix Anal. Appl..

[53]  Luc Van Gool,et al.  A Riemannian Network for SPD Matrix Learning , 2016, AAAI.

[54]  Jonathan H. Manton,et al.  Riemannian Gaussian Distributions on the Space of Symmetric Positive Definite Matrices , 2015, IEEE Transactions on Information Theory.

[55]  Hongdong Li,et al.  Kernel Methods on the Riemannian Manifold of Symmetric Positive Definite Matrices , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[56]  J. Vermeer.  A geometric interpretation of Ungar's addition and of gyration in the hyperbolic plane , 2005 .

[57]  Tat-Seng Chua,et al.  Neural Collaborative Filtering , 2017, WWW.

[58]  Yiqun Liu,et al.  Jointly Learning Explainable Rules for Recommendation with Knowledge Graph , 2019, WWW.

[59]  Christopher R'e,et al.  Low-Dimensional Hyperbolic Knowledge Graph Embeddings , 2020, ACL.

[60]  Qilong Wang,et al.  Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[61]  Vikas Singh,et al.  A Statistical Recurrent Model on the Manifold of Symmetric Positive Definite Matrices , 2018, NeurIPS.

[62]  Praveen Paritosh,et al.  Freebase: a collaboratively created graph database for structuring human knowledge , 2008, SIGMOD Conference.

[63]  Jason Weston,et al.  Translating Embeddings for Modeling Multi-relational Data , 2013, NIPS.

[64]  Pasquale Minervini,et al.  Convolutional 2D Knowledge Graph Embeddings , 2017, AAAI.

[65]  Thomas Hofmann,et al.  Hyperbolic Neural Networks , 2018, NeurIPS.

[66]  Octavian-Eugen Ganea,et al.  Computationally Tractable Riemannian Manifolds for Graph Embeddings , 2020, AAAI.

[67]  Jure Leskovec,et al.  Hyperbolic Graph Convolutional Neural Networks , 2019, NeurIPS.

[68]  Shiguang Shan,et al.  COSONet: Compact Second-Order Network for Video Face Recognition , 2018, ACCV.

[69]  S. Sra.  Positive definite matrices and the S-divergence , 2011, 1110.1773.

[70]  Timothy M. Hospedales,et al.  TuckER: Tensor Factorization for Knowledge Graph Completion , 2019, EMNLP.

[71]  Silvere Bonnabel,et al.  Stochastic Gradient Descent on Riemannian Manifolds , 2011, IEEE Transactions on Automatic Control.

[72]  Cristian Sminchisescu,et al.  Semantic Segmentation with Second-Order Pooling , 2012, ECCV.

[73]  Jure Leskovec,et al.  Hidden factors and hidden topics: understanding rating dimensions with review text , 2013, RecSys.

[74]  David W. Jacobs,et al.  Riemannian Metric Learning for Symmetric Positive Definite Matrices , 2015, ArXiv.

[75]  N. Ayache,et al.  Log‐Euclidean metrics for fast and simple calculus on diffusion tensors , 2006, Magnetic resonance in medicine.

[76]  Matthieu Cord,et al.  Riemannian batch normalization for SPD neural networks , 2019, NeurIPS.

[77]  Shiguang Shan,et al.  Learning Euclidean-to-Riemannian Metric for Point-to-Set Classification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[78]  Junbin Gao,et al.  Kernel Sparse Subspace Clustering on Symmetric Positive Definite Manifolds , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[79]  Pierre Vandergheynst,et al.  Geometric Deep Learning: Going beyond Euclidean data , 2016, IEEE Signal Process. Mag..

[80]  D. Donoho,et al.  Fast Solution of -Norm Minimization Problems When the Solution May Be Sparse , 2008 .

[81]  Timothy M. Hospedales,et al.  Interpreting Knowledge Graph Relation Representation from Word Embeddings , 2021, ICLR.

[82]  Bamdev Mishra,et al.  Manopt, a matlab toolbox for optimization on manifolds , 2013, J. Mach. Learn. Res..

[83]  Zhongmin Shen,et al.  Riemann-Finsler Geometry with Applications to Information Geometry , 2006 .

[84]  Luc Brun,et al.  A Neural Network Based on SPD Manifold Learning for Skeleton-Based Hand Gesture Recognition , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[85]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[86]  Brian C. Lovell,et al.  Convex Class Model on Symmetric Positive Definite Manifolds , 2018, Image Vis. Comput..

[88]  Nicolas Usunier,et al.  Canonical Tensor Decomposition for Knowledge Base Completion , 2018, ICML.

[89]  Shiguang Shan,et al.  Discriminant analysis on Riemannian manifold of Gaussian distributions for face recognition with image sets , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[90]  Fatih Murat Porikli,et al.  Region Covariance: A Fast Descriptor for Detection and Classification , 2006, ECCV.

[91]  Lina Yao,et al.  Quaternion Knowledge Graph Embeddings , 2019, NeurIPS.