Universal Inference Meets Random Projections: A Scalable Test for Log-concavity

Shape constraints yield flexible middle grounds between fully nonparametric and fully parametric approaches to modeling distributions of data. The specific assumption of log-concavity is motivated by applications across economics, survival modeling, and reliability theory. However, there do not currently exist valid tests for whether the underlying density of given data is log-concave. The recent universal likelihood ratio test provides a valid test. The universal test relies on maximum likelihood estimation (MLE), and efficient methods already exist for finding the log-concave MLE. This yields the first test of log-concavity that is provably valid in finite samples in any dimension, for which we also establish asymptotic consistency results. Empirically, we find that the highest power is obtained by using random projections to convert the d-dimensional testing problem into many one-dimensional problems, leading to a simple procedure that is statistically and computationally efficient.

[1]  M. Wand,et al.  Multivariate plug-in bandwidth selection , 1994 .

[2]  Larry Wasserman,et al.  Universal inference , 2019, Proceedings of the National Academy of Sciences.

[3]  Kaspar Rufibach,et al.  Active Set and EM Algorithms for Log-Concave Densities Based on Complete and Censored Data , 2007, 0707.4643.

[4]  James Stephen Marron,et al.  A simple root n bandwidth selector , 1991 .

[5]  Christina Gloeckner,et al.  Modern Applied Statistics With S , 2003 .

[6]  Kohske Takahashi,et al.  Welcome to the Tidyverse , 2019, J. Open Source Softw..

[7]  M. Cule,et al.  Theoretical properties of the log-concave maximum likelihood estimator of a multidimensional density , 2009, 0908.4400.

[8]  A. Genz,et al.  Computation of Multivariate Normal and t Probabilities , 2009 .

[9]  J. Behboodian.  On the Modes of a Mixture of Two Normal Distributions , 1970 .

[10]  A. Bowman.  An alternative method of cross-validation for the smoothing of density estimates , 1984 .

[11]  Luca Scrucca,et al.  mclust 5: Clustering, Classification and Density Estimation Using Gaussian Finite Mixture Models , 2016, R J..

[12]  M. Delignette-Muller,et al.  fitdistrplus: An R Package for Fitting Distributions , 2015 .

[13]  L. Dümbgen,et al.  logcondens: Computations Related to Univariate Log-Concave Density Estimation , 2011 .

[14]  M. Cule,et al.  Maximum likelihood estimation of a multi‐dimensional log‐concave density , 2008, 0804.3989.

[15]  Martin L. Hazelton,et al.  Assessing log-concavity of multivariate densities , 2011 .

[16]  Arlene K. H. Kim,et al.  Global rates of convergence in log-concave density estimation , 2014, 1404.2298.

[17]  R. Koenker,et al.  Shape Constrained Density Estimation Via Penalized Rényi Divergence , 2018, Statistical Science.

[18]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[19]  M. Hazelton,et al.  Cross‐validation Bandwidth Matrices for Multivariate Kernel Density Estimation , 2005 .

[20]  T. Duong,et al.  Multivariate plug-in bandwidth selection with unconstrained pilot bandwidth matrices , 2010 .

[21]  Franz Kappel,et al.  An Implementation of Shor's r-Algorithm , 2000, Comput. Optim. Appl..

[22]  Franziska Wulf,et al.  Minimization Methods For Non Differentiable Functions , 2016 .

[23]  M. Rudemo.  Empirical Choice of Histograms and Kernel Density Estimators , 1982 .

[24]  Robert B. Gramacy,et al.  Maximum likelihood estimation of a multivariate log-concave density , 2010 .

[25]  Alexander Rakhlin,et al.  Optimality of Maximum Likelihood for Log-Concave Density Estimation and Bounded Convex Regression , 2019, 1903.05315.

[26]  M. Bagnoli,et al.  Log-concave probability and its applications , 2004 .

[27]  R. Samworth,et al.  Smoothed log-concave maximum likelihood estimation with applications , 2011, 1102.1191.

[28]  A. Prékopa.  On logarithmic concave measures and functions , 1973 .

[29]  Ralph Roskies,et al.  Bridges: a uniquely flexible HPC resource for new communities and data analytics , 2015, XSEDE.

[30]  M. An.  Log-Concave Probability Distributions: Theory and Statistical Testing , 1996 .

[31]  Nancy Wilkins-Diehr,et al.  XSEDE: Accelerating Scientific Discovery , 2014, Computing in Science & Engineering.

[32]  Rahul Mazumder,et al.  A new computational framework for log-concave density estimation , 2021 .

[33]  W. Wong,et al.  Probability inequalities for likelihood ratios and convergence rates of sieve MLEs , 1995 .

[34]  M. Hazelton,et al.  Plug-in bandwidth matrices for bivariate kernel density estimation , 2003 .

[35]  Raymond J Carroll,et al.  Testing and Estimating Shape-Constrained Nonparametric Density and Regression in the Presence of Measurement Error , 2011, Journal of the American Statistical Association.