Hyperbolic Neural Networks++

Hyperbolic spaces, which have the capacity to embed tree structures without distortion owing to their exponential volume growth, have recently been applied to machine learning to better capture the hierarchical nature of data. In this study, we reconsider a way to generalize the fundamental components of neural networks in a single hyperbolic geometry model, and propose novel methodologies to construct a multinomial logistic regression, fully-connected layers, convolutional layers, and attention mechanisms under a unified mathematical interpretation, without increasing the parameters. A series of experiments show the parameter efficiency of our methods compared to a conventional hyperbolic component, and stability and outperformance over their Euclidean counterparts.

[1]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[2]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[3]  Douwe Kiela,et al.  Hyperbolic Graph Neural Networks , 2019, NeurIPS.

[4]  Douwe Kiela,et al.  Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry , 2018, ICML.

[5]  Yee Whye Teh,et al.  Continuous Hierarchical Representations with Poincaré Variational Auto-Encoders , 2019, NeurIPS.

[6]  Thomas Hofmann,et al.  Hyperbolic Entailment Cones for Learning Hierarchical Embeddings , 2018, ICML.

[7]  A. Ungar.  Beyond the Einstein Addition Law and its Gyroscopic Thomas Precession: The Theory of Gyrogroups and Gyrovector Spaces , 2001 .

[8]  Bonnie Berger,et al.  Large-Margin Classification in Hyperbolic Space , 2018, AISTATS.

[9]  Ivan Ovinnikov,et al.  Poincaré Wasserstein Autoencoder , 2019, ArXiv.

[10]  Abraham Albert Ungar,et al.  A Gyrovector Space Approach to Hyperbolic Geometry , 2009, A Gyrovector Space Approach to Hyperbolic Geometry.

[11]  Ramana Rao,et al.  A focus+context technique based on hyperbolic geometry for visualizing large hierarchies , 1995, CHI '95.

[12]  Tobias Friedrich,et al.  Efficient Embedding of Scale-Free Graphs in the Hyperbolic Plane , 2018, IEEE/ACM Transactions on Networking.

[13]  M. Spivak.  A comprehensive introduction to differential geometry , 1979 .

[14]  Amin Vahdat,et al.  On curvature and temperature of complex networks , 2009, Physical review. E, Statistical, nonlinear, and soft matter physics.

[15]  Christopher Hopper,et al.  The Ricci Flow in Riemannian Geometry: A Complete Proof of the Differentiable 1/4-Pinching Sphere Theorem , 2010 .

[16]  Douwe Kiela,et al.  Poincaré Embeddings for Learning Hierarchical Representations , 2017, NIPS.

[17]  Philipp Koehn,et al.  Findings of the 2017 Conference on Machine Translation (WMT17) , 2017, WMT.

[18]  Christopher De Sa,et al.  Representation Tradeoffs for Hyperbolic Embeddings , 2018, ICML.

[19]  Razvan Pascanu,et al.  Hyperbolic Attention Networks , 2018, ICLR.

[20]  Shoichiro Yamaguchi,et al.  A Wrapped Normal Distribution on Hyperbolic Space for Gradient-Based Learning , 2019, ICML.

[21]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[22]  Rasul Karimov,et al.  Geoopt: Riemannian Optimization in PyTorch , 2020, ArXiv.

[23]  Kaoru Katayama,et al.  Indexing Method for Hierarchical Graphs based on Relation among Interlacing Sequences of Eigenvalues , 2015, J. Inf. Process..

[24]  M. Newman.  Power laws, Pareto distributions and Zipf's law , 2005 .

[25]  Yann Dauphin,et al.  Convolutional Sequence to Sequence Learning , 2017, ICML.

[26]  Gary Bécigneul,et al.  Poincaré GloVe: Hyperbolic Word Embeddings , 2018, ICLR.

[27]  Thomas Hofmann,et al.  Hyperbolic Neural Networks , 2018, NeurIPS.

[28]  A. Ungar.  Analytic Hyperbolic Geometry and Albert Einstein's Special Theory of Relativity , 2008 .

[29]  Amin Vahdat,et al.  Hyperbolic Geometry of Complex Networks , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[30]  Octavian-Eugen Ganea,et al.  Mixed-curvature Variational Autoencoders , 2019, ICLR.

[31]  Myle Ott,et al.  fairseq: A Fast, Extensible Toolkit for Sequence Modeling , 2019, NAACL.

[32]  A. Ungar.  Hyperbolic trigonometry and its application in the Poincaré ball model of hyperbolic geometry , 2001 .

[33]  Renjie Liao,et al.  Lorentzian Distance Learning for Hyperbolic Representations , 2019, ICML.

[34]  Marko Valentin Micic,et al.  Hyperbolic Deep Learning for Chinese Natural Language Understanding , 2018, ArXiv.

[35]  Timothy M. Hospedales,et al.  Multi-relational Poincaré Graph Embeddings , 2019, NeurIPS.

[36]  Frederic Sala,et al.  Learning Mixed-Curvature Representations in Product Spaces , 2018, ICLR.

[37]  Jure Leskovec,et al.  Hyperbolic Graph Convolutional Neural Networks , 2019, NeurIPS.

[38]  Yee Whye Teh,et al.  Set Transformer , 2018, ICML.

[39]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[40]  Gary Bécigneul,et al.  Riemannian Adaptive Optimization Methods , 2018, ICLR.

[41]  John D. Lafferty,et al.  Hyperplane margin classifiers on the multinomial manifold , 2004, ICML.

[42]  A. Ungar,et al.  Einstein's Special Relativity: The Hyperbolic Geometric Viewpoint , 2013, 1302.6961.