A new computational framework for log-concave density estimation

In Statistics, log-concave density estimation is a central problem within the field of nonparametric inference under shape constraints. Despite great progress in recent years on the statistical theory of the canonical estimator, namely the log-concave maximum likelihood estimator, adoption of this method has been hampered by the complexities of the non-smooth convex optimization problem that underpins its computation. We provide enhanced understanding of the structural properties of this optimization problem, which motivates the proposal of new algorithms, based on both randomized and Nesterov smoothing, combined with an appropriate integral discretization of increasing accuracy. We prove that these methods enjoy, both with high probability and in expectation, a convergence rate of order 1/T up to logarithmic factors on the objective function scale, where T denotes the number of iterations. The benefits of our new computational framework are demonstrated on both synthetic and real data, and our implementation is available in a github repository LogConcComp (Log-Concave Computation).

[1]  J. Wellner,et al.  Bounding distributional errors via density ratios , 2019, Bernoulli.

[2]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[3]  Fabian Rathke,et al.  Fast multivariate log-concave density estimation , 2018, Comput. Stat. Data Anal..

[4]  Panos M. Pardalos,et al.  Convex optimization theory , 2010, Optim. Methods Softw..

[5]  Yurii Nesterov,et al.  Primal-dual subgradient methods for convex problems , 2005, Math. Program..

[6]  Franz Kappel,et al.  An Implementation of Shor's r-Algorithm , 2000, Comput. Optim. Appl..

[7]  R. Koenker,et al.  QUASI-CONCAVE DENSITY ESTIMATION , 2010, 1007.4013.

[8]  Qiyang Han,et al.  APPROXIMATION AND ESTIMATION OF s-CONCAVE DENSITIES VIA RÉNYI DIVERGENCES. , 2015, Annals of statistics.

[9]  M. Cule,et al.  Theoretical properties of the log-concave maximum likelihood estimator of a multidimensional density , 2009, 0908.4400.

[10]  Sabyasachi Chatterjee,et al.  Isotonic regression in general dimensions , 2017, The Annals of Statistics.

[11]  R. Barber,et al.  Local continuity of log-concave projection, with applications to estimation under model misspecification , 2020, 2002.06117.

[12]  M. Yuan,et al.  Independent component analysis via nonparametric maximum likelihood estimation , 2012, 1206.0457.

[13]  L. Duembgen,et al.  Maximum likelihood estimation of a log-concave density and its distribution function: Basic properties and uniform consistency , 2007, 0709.0334.

[14]  Adityanand Guntuboyina,et al.  On risk bounds in isotonic and other shape restricted regression problems , 2013, 1311.3765.

[15]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[16]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[17]  Lutz Duembgen,et al.  On an Auxiliary Function for Log-Density Estimation , 2008, 0807.4719.

[18]  R. Samworth,et al.  Isotonic regression with unknown permutations: Statistics, computation, and adaptation , 2020, The Annals of Statistics.

[19]  C. Hildreth.  Point Estimates of Ordinates of Concave Functions , 1954 .

[20]  T. Cai,et al.  A Framework For Estimation of Convex Functions , 2015 .

[21]  V. Buldygin,et al.  Metric characterization of random variables and random processes , 2000 .

[22]  Rina Foygel Barber,et al.  Contraction and uniform convergence of isotonic regression , 2017, Electronic Journal of Statistics.

[23]  L. Dümbgen,et al.  logcondens: Computations Related to Univariate Log-Concave Density Estimation , 2011 .

[24]  Arlene K. H. Kim,et al.  Adaptation in log-concave density estimation , 2016, The Annals of Statistics.

[25]  Arlene K. H. Kim,et al.  Adaptation in multivariate log-concave density estimation , 2018, The Annals of Statistics.

[26]  Y. Nesterov.  Primal-Dual Subgradient Methods for Convex Problems , 2005 .

[27]  Arlene K. H. Kim,et al.  Global rates of convergence in log-concave density estimation , 2014, 1404.2298.

[28]  G. Walther.  Detecting the Presence of Mixing with Multiscale Maximum Likelihood , 2002 .

[29]  W. Gilks,et al.  Adaptive rejection sampling from log-concave density functions , 1993 .

[30]  Jeremy Kepner,et al.  Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis , 2018, 2018 IEEE High Performance extreme Computing Conference (HPEC).

[31]  Geurt Jongbloed,et al.  Nonparametric Estimation under Shape Constraints , 2014 .

[32]  David P. Dobkin,et al.  The quickhull algorithm for convex hulls , 1996, TOMS.

[33]  Franziska Wulf,et al.  Minimization Methods For Non Differentiable Functions , 2016 .

[34]  Ilias Diakonikolas,et al.  Near-Optimal Sample Complexity Bounds for Maximum Likelihood Estimation of Multivariate Log-concave Densities , 2018, COLT.

[35]  Cun-Hui Zhang.  Risk bounds in isotonic regression , 2002 .

[36]  G. Walther.  Inference and Modeling with Log-concave Distributions , 2009, 1010.0305.

[37]  N. Shor.  Nondifferentiable Optimization and Polynomial Problems , 1998 .

[38]  Martin J. Wainwright,et al.  Randomized Smoothing for Stochastic Optimization , 2011, SIAM J. Optim..

[39]  Adityanand Guntuboyina,et al.  Global risk bounds and adaptation in univariate convex regression , 2013, 1305.1648.

[40]  J. Wellner,et al.  ST ] 2 5 Ja n 20 16 MULTIVARIATE CONVEX REGRESSION : GLOBAL RISK BOUNDS AND ADAPTATION By Qiyang Han , 2016 .

[41]  Charles R. Doss,et al.  GLOBAL RATES OF CONVERGENCE OF THE MLES OF LOG-CONCAVE AND s-CONCAVE DENSITIES. , 2013, Annals of statistics.

[42]  A. Dalalyan.  Theoretical guarantees for approximate sampling from smooth and log‐concave densities , 2014, 1412.7392.

[43]  Nicholas G. Polson,et al.  Sampling from log-concave distributions , 1994 .

[44]  F. Clarke.  Optimization And Nonsmooth Analysis , 1983 .

[45]  R. Samworth.  Recent Progress in Log-Concave Density Estimation , 2017, Statistical Science.

[46]  E. Seijo,et al.  Nonparametric Least Squares Estimation of a Multivariate Convex Regression Function , 2010, 1003.4765.

[47]  L. Duembgen,et al.  APPROXIMATION BY LOG-CONCAVE DISTRIBUTIONS, WITH APPLICATIONS TO REGRESSION , 2010, 1002.3448.

[48]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[49]  Kazuoki Azuma.  WEIGHTED SUMS OF CERTAIN DEPENDENT RANDOM VARIABLES , 1967 .

[50]  Robert B. Gramacy,et al.  Maximum likelihood estimation of a multivariate log-concave density , 2010 .

[51]  Ilias Diakonikolas,et al.  A Polynomial Time Algorithm for Log-Concave Maximum Likelihood via Locally Exponential Families , 2019, NeurIPS.

[52]  D. Rudolf,et al.  Hit-and-Run for Numerical Integration , 2012, 1212.4486.

[53]  W. Chan,et al.  Unimodality, convexity, and applications , 1989 .

[54]  Daniela Pucci de Farias,et al.  Decentralized Resource Allocation in Dynamic Networks of Agents , 2008, SIAM J. Optim..

[55]  Yong Wang,et al.  A fast algorithm for univariate log‐concave density estimation , 2018, Australian & New Zealand Journal of Statistics.

[56]  S. Geer,et al.  Multivariate log-concave distributions as a nearly parametric model , 2008, Am. Math. Mon..

[57]  Bodhisattva Sen,et al.  Editorial: Special Issue on “Nonparametric Inference Under Shape Constraints” , 2018, Statistical Science.

[58]  Yuval Dagan,et al.  The Log-Concave Maximum Likelihood Estimator is Optimal in High Dimensions , 2019, ArXiv.

[59]  Jon A Wellner,et al.  NONPARAMETRIC ESTIMATION OF MULTIVARIATE CONVEX-TRANSFORMED DENSITIES. , 2009, Annals of statistics.

[60]  R. Samworth,et al.  High-dimensional nonparametric density estimation via symmetry and shape constraints , 2019, The Annals of Statistics.

[61]  H. D. Brunk,et al.  Statistical inference under order restrictions : the theory and application of isotonic regression , 1973 .

[62]  P. McMullen.  The maximum numbers of faces of a convex polytope , 1970 .

[63]  Hendrik P. Lopuhaä,et al.  Limit Theory in Monotone Function Estimation , 2018, Statistical Science.

[64]  Qiyang Han,et al.  Global empirical risk minimizers with "shape constraints" are rate optimal in general dimensions , 2019, 1905.12823.

[65]  Adityanand Guntuboyina,et al.  On the risk of convex-constrained least squares estimators under misspecification , 2017, Bernoulli.

[66]  M. Cule,et al.  Maximum likelihood estimation of a multi‐dimensional log‐concave density , 2008, 0804.3989.

[67]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[68]  Jayanta Kumar Pal,et al.  Estimating a Polya Frequency Function , 2006 .

[69]  Santosh S. Vempala,et al.  Fast Algorithms for Logconcave Functions: Sampling, Rounding, Integration and Optimization , 2006, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06).

[70]  R. Samworth,et al.  Generalized additive and index models with shape constraints , 2014, 1404.2957.

[71]  Angelia Nedic,et al.  On stochastic gradient and subgradient methods with adaptive steplength sequences , 2011, Autom..

[72]  H. M. Möller,et al.  Invariant Integration Formulas for the n-Simplex by Combinatorial Methods , 1978 .

[73]  Lin Xiao,et al.  Dual Averaging Methods for Regularized Stochastic Learning and Online Optimization , 2009, J. Mach. Learn. Res..

[74]  Martin J. Wainwright,et al.  Information-Theoretic Lower Bounds on the Oracle Complexity of Stochastic Convex Optimization , 2010, IEEE Transactions on Information Theory.

[75]  R. Samworth,et al.  Smoothed log-concave maximum likelihood estimation with applications , 2011, 1102.1191.

[76]  P. Bellec.  Sharp oracle inequalities for Least Squares estimators in shape restricted regression , 2015, 1510.08029.

[77]  Achim Klenke,et al.  Probability theory - a comprehensive course , 2008, Universitext.

[78]  U. Grenander.  On the theory of mortality measurement , 1956 .