Flexible Modeling of Diversity with Strongly Log-Concave Distributions

Strongly log-concave (SLC) distributions are a rich class of discrete probability distributions over subsets of some ground set. They are strictly more general than strongly Rayleigh (SR) distributions such as the well-known determinantal point process. While SR distributions offer elegant models of diversity, they lack an easy control over how they express diversity. We propose SLC as the right extension of SR that enables easier, more intuitive control over diversity, illustrating this via examples of practical importance. We develop two fundamental tools needed to apply SLC distributions to learning and inference: sampling and mode finding. For sampling we develop an MCMC sampler and give theoretical mixing time bounds. For mode finding, we establish a weak log-submodularity property for SLC functions and derive optimization guarantees for a distorted greedy algorithm.

[1]  Hui Lin,et al.  Learning Mixtures of Submodular Shells with Application to Document Summarization , 2012, UAI.

[2]  Joseph Naor,et al.  A Tight Linear Time (1/2)-Approximation for Unconstrained Submodular Maximization , 2012, 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science.

[3]  P. Diaconis,et al.  LOGARITHMIC SOBOLEV INEQUALITIES FOR FINITE MARKOV CHAINS , 1996 .

[4]  Rishabh K. Iyer,et al.  Submodular Point Processes with Applications to Machine learning , 2015, AISTATS.

[5]  Suvrit Sra,et al.  Elementary Symmetric Polynomials for Optimal Experimental Design , 2017, NIPS.

[6]  Emil Jerábek,et al.  Dual weak pigeonhole principle, Boolean complexity, and derandomization , 2004, Annals of Pure and Applied Logic.

[7]  Manfred K. Warmuth,et al.  Unbiased estimates for linear regression via volume sampling , 2017, NIPS.

[8]  A Randomized Coordinate Descent Method with Volume Sampling , 2020, SIAM J. Optim..

[9]  Jasper Snoek,et al.  DPPNet: Approximating Determinantal Point Processes with Deep Networks , 2019, NeurIPS.

[10]  Nima Anari,et al.  Monte Carlo Markov Chain Algorithms for Sampling Strongly Rayleigh Distributions and Determinantal Point Processes , 2016, COLT.

[11]  Suvrit Sra,et al.  Kronecker Determinantal Point Processes , 2016, NIPS.

[12]  Andreas Krause,et al.  From MAP to Marginals: Variational Inference in Bayesian Submodular Models , 2014, NIPS.

[13]  Karim A. Adiprasito,et al.  Hodge theory for combinatorial geometries , 2015, Annals of Mathematics.

[14]  P. Diaconis,et al.  Geometric Bounds for Eigenvalues of Markov Chains , 1991 .

[15]  Suvrit Sra,et al.  Exponentiated Strongly Rayleigh Distributions , 2018, NeurIPS.

[16]  Nima Anari,et al.  Log-Concave Polynomials III: Mason's Ultra-Log-Concavity Conjecture for Independent Sets of Matroids , 2018, ArXiv.

[17]  Leonid Gurvits,et al.  On multivariate Newton-like inequalities , 2008, 0812.3687.

[18]  Suvrit Sra,et al.  Fixed-point algorithms for learning determinantal point processes , 2015, ICML.

[19]  Suvrit Sra,et al.  Polynomial time algorithms for dual volume sampling , 2017, NIPS.

[20]  Andrew Gelman,et al.  General methods for monitoring convergence of iterative simulations , 1998 .

[21]  Ryan P. Adams,et al.  Priors for Diversity in Generative Latent Variable Models , 2012, NIPS.

[22]  T. Raghavan,et al.  Nonnegative Matrices and Applications , 1997 .

[23]  Alexandros G. Dimakis,et al.  Scalable Greedy Feature Selection via Weak Submodularity , 2017, AISTATS.

[24]  Ben Taskar,et al.  Near-Optimal MAP Inference for Determinantal Point Processes , 2012, NIPS.

[25]  Heng Guo,et al.  Modified log-Sobolev Inequalities for Strongly Log-Concave Distributions , 2019, 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS).

[26]  Ben Taskar,et al.  k-DPPs: Fixed-Size Determinantal Point Processes , 2011, ICML.

[27]  Ben Taskar,et al.  Determinantal Point Processes for Machine Learning , 2012, Found. Trends Mach. Learn..

[28]  Francis R. Bach,et al.  Learning Determinantal Point Processes in Sublinear Time , 2016, AISTATS.

[29]  Alkis Gotovos,et al.  Sampling from Probabilistic Submodular Models , 2015, NIPS.

[30]  Suvrit Sra,et al.  Fast DPP Sampling for Nystrom with Application to Kernel Methods , 2016, ICML.

[31]  Suvrit Sra,et al.  Fast Mixing Markov Chains for Strongly Rayleigh Measures, DPPs, and Constrained Sampling , 2016, NIPS.

[32]  T. Liggett,et al.  Negative dependence and the geometry of polynomials , 2007, 0707.2340.

[33]  Suvrit Sra,et al.  Diversity Networks: Neural Network Compression Using Determinantal Point Processes , 2015, 1511.05077.

[34]  Nisheeth K. Vishnoi,et al.  Fair and Diverse DPP-based Data Summarization , 2018, ICML.

[35]  Ulrich Paquet,et al.  Low-Rank Factorization of Determinantal Point Processes , 2017, AAAI.

[36]  Alkis Gotovos,et al.  Strong Log-Concavity Does Not Imply Log-Submodularity , 2019, ArXiv.

[37]  P. Brand'en.  Polynomials with the half-plane property and matroid theory , 2006, math/0605678.

[38]  Abhimanyu Das,et al.  Submodular meets Spectral: Greedy Algorithms for Subset Selection, Sparse Approximation and Dictionary Selection , 2011, ICML.

[39]  Andreas Krause,et al.  Scalable Variational Inference in Log-supermodular Models , 2015, ICML.

[40]  Malik Magdon-Ismail,et al.  Exponential Inapproximability of Selecting a Maximum Volume Sub-matrix , 2011, Algorithmica.

[41]  Amin Karbasi,et al.  Submodular Maximization Beyond Non-negativity: Guarantees, Fast Algorithms, and Applications , 2019, ICML.

[42]  Andreas Krause,et al.  Provable Variational Inference for Constrained Log-Submodular Models , 2018, NeurIPS.

[43]  Nima Anari,et al.  Log-concave polynomials II: high-dimensional walks and an FPRAS for counting bases of a matroid , 2018, STOC.

[44]  Moran Feldman,et al.  Guess Free Maximization of Submodular and Linear Sums , 2018, Algorithmica.

[45]  C. Berg,et al.  Harmonic Analysis on Semigroups: Theory of Positive Definite and Related Functions , 1984 .

[46]  Y. Peres,et al.  Determinantal Processes and Independence , 2005, math/0503110.

[47]  Nima Anari,et al.  Log-Concave Polynomials, Entropy, and a Deterministic Approximation Algorithm for Counting Bases of Matroids , 2018, 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS).

[48]  June Huh,et al.  COMBINATORIAL APPLICATIONS OF THE HODGE–RIEMANN RELATIONS , 2017, Proceedings of the International Congress of Mathematicians (ICM 2018).