A Polynomial Time Algorithm for Maximum Likelihood Estimation of Multivariate Log-concave Densities

We study the problem of computing the maximum likelihood estimator (MLE) of multivariate log-concave densities. Our main result is the first computationally efficient algorithm for this problem. In more detail, we give an algorithm that, on input a set of $n$ points in $\mathbb{R}^d$ and an accuracy parameter $\epsilon>0$, it runs in time $\text{poly}(n, d, 1/\epsilon)$, and outputs a log-concave density that with high probability maximizes the log-likelihood up to an additive $\epsilon$. Our approach relies on a natural convex optimization formulation of the underlying problem that can be efficiently solved by a projected stochastic subgradient method. The main challenge lies in showing that a stochastic subgradient of our objective function can be efficiently approximated. To achieve this, we rely on structural results on approximation of log-concave densities and leverage classical algorithmic tools on volume approximation of convex bodies and uniform sampling from convex sets.

[1]  H. D. Brunk On the Estimation of Parameters Restricted by Inequalities , 1958 .

[2]  Santosh S. Vempala,et al.  Simulated annealing in convex bodies and an O*(n4) volume algorithm , 2006, J. Comput. Syst. Sci..

[3]  Carl M. O’Brien,et al.  Nonparametric Estimation under Shape Constraints: Estimators, Algorithms and Asymptotics , 2016 .

[4]  D. L. Hanson,et al.  Consistency in Concave Regression , 1976 .

[5]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[6]  R. Stanley Log‐Concave and Unimodal Sequences in Algebra, Combinatorics, and Geometry a , 1989 .

[7]  Rocco A. Servedio,et al.  Near-Optimal Density Estimation in Near-Linear Time Using Variable-Width Histograms , 2014, NIPS.

[8]  Daniel M. Kane,et al.  Optimal Learning via the Fourier Transform for Sums of Independent Integer Random Variables , 2015, COLT.

[9]  S. Vempala,et al.  Hit-and-Run from a Corner , 2006 .

[10]  Fadoua Balabdaoui,et al.  Estimation of a k‐monotone density: characterizations, consistency and minimax lower bounds , 2010, Statistica Neerlandica.

[11]  E. Wegman Maximum likelihood estimation of a unimodal density. II , 1970 .

[12]  Arlene K. H. Kim,et al.  Global rates of convergence in log-concave density estimation , 2014, 1404.2298.

[13]  M. Cule,et al.  Maximum likelihood estimation of a multi‐dimensional log‐concave density , 2008, 0804.3989.

[14]  P. Groeneboom Estimating a monotone density , 1984 .

[15]  Charles R. Doss,et al.  GLOBAL RATES OF CONVERGENCE OF THE MLES OF LOG-CONCAVE AND s-CONCAVE DENSITIES. , 2013, Annals of statistics.

[16]  Luc Devroye,et al.  Combinatorial methods in density estimation , 2001, Springer series in statistics.

[17]  Rocco A. Servedio,et al.  Density estimation for shift-invariant multidimensional distributions , 2018, ITCS.

[18]  Ilias Diakonikolas,et al.  Sample-Optimal Density Estimation in Nearly-Linear Time , 2015, SODA.

[19]  S. Vempala,et al.  The geometry of logconcave functions and sampling algorithms , 2007 .

[20]  G. Walther Inference and Modeling with Log-concave Distributions , 2009, 1010.0305.

[21]  L. Duembgen,et al.  Maximum likelihood estimation of a log-concave density and its distribution function: Basic properties and uniform consistency , 2007, 0709.0334.

[22]  Chinmay Hegde,et al.  Fast and Near-Optimal Algorithms for Approximating Distributions by Histograms , 2015, PODS.

[23]  Ryan O'Donnell,et al.  Learning Sums of Independent Integer Random Variables , 2013, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science.

[24]  Daniel M. Kane,et al.  Efficient Robust Proper Learning of Log-concave Distributions , 2016, ArXiv.

[25]  L. Birge On the Risk of Histograms for Estimating Decreasing Densities , 1987 .

[26]  Qiyang Han,et al.  APPROXIMATION AND ESTIMATION OF s-CONCAVE DENSITIES VIA RÉNYI DIVERGENCES. , 2015, Annals of statistics.

[27]  M. Cule,et al.  Theoretical properties of the log-concave maximum likelihood estimator of a multidimensional density , 2009, 0908.4400.

[28]  Anindya De,et al.  A size-free CLT for poisson multinomials and its applications , 2015, STOC.

[29]  Bernd Sturmfels,et al.  Geometry of Log-Concave Density Estimation , 2017, Discrete & Computational Geometry.

[30]  Arlene K. H. Kim,et al.  Adaptation in log-concave density estimation , 2016, The Annals of Statistics.

[31]  Rocco A. Servedio,et al.  Learning k-Modal Distributions via Testing , 2012, Theory Comput..

[32]  Rocco A. Servedio,et al.  Learning mixtures of structured distributions over discrete domains , 2012, SODA.

[33]  Jon A Wellner,et al.  Estimation of a discrete monotone distribution. , 2009, Electronic journal of statistics.

[34]  M. An Log-Concave Probability Distributions: Theory and Statistical Testing , 1996 .

[35]  R. Koenker,et al.  QUASI-CONCAVE DENSITY ESTIMATION , 2010, 1007.4013.

[36]  J. Wellner,et al.  Estimation of a k-monotone density: limit distribution theory and the Spline connection , 2005, math/0509081.

[37]  J. Wellner,et al.  On the rate of convergence of the maximum likelihood estimator of a k-monotone density , 2009 .

[38]  Daniel M. Kane,et al.  Learning Multivariate Log-concave Distributions , 2016, COLT.

[39]  Rocco A. Servedio,et al.  Learning Poisson Binomial Distributions , 2011, STOC '12.

[40]  Gregory Valiant,et al.  An Efficient Algorithm for High-Dimensional Log-Concave Maximum Likelihood , 2018, ArXiv.

[41]  Anne-Laure Fougères,et al.  Estimation de densités unimodales , 1997 .

[42]  J. M. Bremner,et al.  Statistical Inference under Restrictions , 1973 .

[43]  Daniel M. Kane,et al.  The fourier transform of poisson multinomial distributions and its algorithmic applications , 2015, STOC.

[44]  Prakasa Rao Estimation of a unimodal density , 1969 .

[45]  Daniel M. Kane,et al.  Properly Learning Poisson Binomial Distributions in Almost Polynomial Time , 2015, COLT.

[46]  John C. Duchi Introductory lectures on stochastic optimization , 2018, IAS/Park City Mathematics Series.

[47]  A. U.S.,et al.  Testing for multimodality with dependent data , 2004 .

[48]  L. Birge Estimating a Density under Order Restrictions: Nonasymptotic Minimax Risk , 1987 .

[49]  R. Samworth,et al.  Smoothed log-concave maximum likelihood estimation with applications , 2011, 1102.1191.

[50]  Fabian Rathke,et al.  Fast multivariate log-concave density estimation , 2018, Comput. Stat. Data Anal..

[51]  J. Wellner,et al.  Limit Distribution Theory for Maximum Likelihood Estimation of a Log-Concave Density. , 2007, Annals of statistics.

[52]  M. Bagnoli,et al.  Log-concave probability and its applications , 2004 .

[53]  U. Grenander On the theory of mortality measurement , 1956 .

[54]  Fadoua Balabdaoui,et al.  Inference for a mixture of symmetric distributions under log-concavity , 2014, 1411.4708.

[55]  R. Samworth Recent Progress in Log-Concave Density Estimation , 2017, Statistical Science.

[56]  Ilias Diakonikolas,et al.  Near-Optimal Sample Complexity Bounds for Maximum Likelihood Estimation of Multivariate Log-concave Densities , 2018, COLT.

[57]  Jerry Li,et al.  Fast and Sample Near-Optimal Algorithms for Learning Multidimensional Histograms , 2018, COLT.

[58]  Ronitt Rubinfeld,et al.  Testing Shape Restrictions of Discrete Distributions , 2015, Theory of Computing Systems.

[59]  J. Wellner,et al.  Log-Concavity and Strong Log-Concavity: a review. , 2014, Statistics surveys.

[60]  Miklós Simonovits,et al.  Random walks and an O*(n5) volume algorithm for convex bodies , 1997, Random Struct. Algorithms.

[61]  Rocco A. Servedio,et al.  Explorer Efficient Density Estimation via Piecewise Polynomial Approximation , 2013 .