Metric Space Magnitude for Evaluating the Diversity of Latent Representations

The magnitude of a metric space is a recently-established invariant, providing a measure of the 'effective size' of a space across multiple scales while also capturing numerous geometrical properties. We develop a family of magnitude-based measures of the intrinsic diversity of latent representations, formalising a novel notion of dissimilarity between magnitude functions of finite metric spaces. Our measures are provably stable under perturbations of the data, can be efficiently calculated, and enable a rigorous multi-scale comparison of latent representations. We show the utility and superior performance of our measures in an experimental suite that comprises different domains and tasks, including the evaluation of diversity, the detection of mode collapse, and the evaluation of generative models for text, image, and graph data.

[1]  Emily Roff,et al.  The small-scale limit of magnitude and the one-point property , 2023, 2312.14497.

[2]  Bastian Alexander Rieck,et al.  Metric Space Magnitude and Generalisation in Neural Networks , 2023, TAG-ML.

[3]  Adji B. Dieng,et al.  The Vendi Score: A Diversity Evaluation Metric for Machine Learning , 2022, ArXiv.

[4]  Guido Montúfar,et al.  On the effectiveness of persistent homology , 2022, NeurIPS.

[5]  Steve Huntsman,et al.  Parallel black-box optimization of expensive high-dimensional multimodal functions via magnitude , 2022, ArXiv.

[6]  Steve Huntsman.  Diversity Enhancement via Magnitude , 2022, EMO.

[7]  Graham W. Taylor,et al.  On Evaluation Metrics for Graph Generative Models , 2022, ICLR.

[8]  Karsten M. Borgwardt,et al.  The magnitude vector of images , 2021, ArXiv.

[9]  Karsten M. Borgwardt,et al.  Evaluation Metrics for Graph Generative Models: Problems, Pitfalls, and Practical Solutions , 2021, ICLR.

[10]  T. Leinster.  Entropy and Diversity: The Axiomatic Approach , 2020, ArXiv.

[11]  Glenn Fung,et al.  Weighting vectors for machine learning: numerical harmonic analysis applied to boundary detection , 2020, ArXiv.

[12]  Elizabeth Clark,et al.  Evaluation of Text Generation: A Survey , 2020, ArXiv.

[13]  Jonathan Berant,et al.  Evaluating the Evaluation of Diversity in Natural Language Generation , 2020, EACL.

[14]  Yi Zhang,et al.  Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collections , 2020, LREC.

[15]  Seong Joon Oh,et al.  Reliable Fidelity and Diversity Metrics for Generative Models , 2020, ICML.

[16]  Dejan Govc,et al.  Persistent magnitude , 2019, Journal of Pure and Applied Algebra.

[17]  Iryna Gurevych,et al.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks , 2019, EMNLP.

[18]  Alan W. Black,et al.  Boosting Dialog Response Generation , 2019, ACL.

[19]  Peter Bubenik,et al.  Persistent homology detects curvature , 2019, Inverse Problems.

[20]  Julien Rabin,et al.  Revisiting precision recall definition for generative modeling , 2019, ICML.

[21]  Jaakko Lehtinen,et al.  Improved Precision and Recall Metric for Assessing Generative Models , 2019, NeurIPS.

[22]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[23]  Bernard De Baets,et al.  Ecological Diversity: Measuring the Unmeasurable , 2018, Mathematics.

[24]  Olivier Bachem,et al.  Assessing Generative Models via Precision and Recall , 2018, NeurIPS.

[25]  Jure Leskovec,et al.  GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models , 2018, ICML.

[26]  Michael Shulman,et al.  Magnitude homology of enriched categories and metric spaces , 2017, Algebraic & Geometric Topology.

[27]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Matthias Bethge,et al.  A note on the evaluation of generative models , 2015, ICLR.

[29]  Søren Hauberg,et al.  Geodesic exponential kernels: When curvature and linearity conflict , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  M. Meckes.  Magnitude, Diversity, Capacities, and Dimensions of Metric Spaces , 2013, 1308.5407.

[31]  Facundo Mémoli,et al.  Some Properties of Gromov–Hausdorff Distances , 2012, Discret. Comput. Geom..

[32]  T. Leinster.  The magnitude of metric spaces , 2010, Documenta Mathematica.

[33]  Mark W. Meckes,et al.  Positive definite metric spaces , 2010, 1012.5863.

[34]  Bernhard Schölkopf,et al.  A Kernel Method for the Two-Sample-Problem , 2006, NIPS.

[35]  Nicholas J. Higham,et al.  Cholesky factorization , 2009 .

[36]  A. Solow,et al.  Measuring biological diversity , 2006, Environmental and Ecological Statistics.

[37]  Nonnegative matrices. , 2022 .