Methods for Shape-Constrained Kernel Density Estimation

Nonparametric density estimators are used to estimate an unknown probability density while making minimal assumptions about its functional form. Although the low reliance of nonparametric estimators on modelling assumptions is a benefit, their performance will be improved if auxiliary information about the density’s shape is incorporated into the estimate. Auxiliary information can take the form of shape constraints, such as unimodality or symmetry, that the estimate must satisfy. Finding the constrained estimate is usually a difficult optimization problem, however, and a consistent framework for finding estimates across a variety of problems is lacking. It is proposed to find shape-constrained density estimates by starting with a pilot estimate obtained by standard methods, and subsequently adjusting its shape until the constraints are satisfied. This strategy is part of a general approach, in which a constrained estimation problem is defined by an estimator, a method of shape adjustment, a constraint, and an objective function. Optimization methods are developed to suit this approach, with a focus on kernel density estimation under a variety of constraints. Two methods of shape adjustment are examined in detail. The first is data sharpening, for which two optimization algorithms are proposed: a greedy algorithm that runs quickly but can handle a limited set of constraints, and a particle swarm algorithm that is suitable for a wider range of problems. The second is the method of adjustment curves, for which it is often possible to use quadratic programming to find optimal estimates. The methods presented here can be used for univariate or higher-dimensional kernel density estimation with shape constraints. They can also be extended to other estimators, in both the density estimation and regression settings. As such they constitute a step toward a truly general optimizer, that can be used on arbitrary combinations of estimator and constraint.

[1]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[2]  Mary C. Meyer.  INFERENCE USING SHAPE-RESTRICTED REGRESSION SPLINES , 2008, 0811.1705.

[3]  C. JonesM.,et al.  Maximum likelihood kernel density estimation , 2009 .

[4]  Andries Petrus Engelbrecht,et al.  A new particle swarm optimiser for linearly constrained optimisation , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[5]  T. Gasser,et al.  Nonparametric Density Estimation under Unimodality and Monotonicity Constraints , 1999 .

[6]  Peter Hall,et al.  High order data sharpening for density estimation , 2002 .

[7]  Jussi Klemel,et al.  Smoothing of Multivariate Data , 2009 .

[8]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[9]  M. C. Jones,et al.  Maximum likelihood kernel density estimation , 2005 .

[10]  Andreas Antoniou,et al.  Practical Optimization: Algorithms and Engineering Applications , 2007, Texts in Computer Science.

[11]  El-Ghazali Talbi,et al.  Metaheuristics - From Design to Implementation , 2009 .

[12]  J. Kennedy,et al.  Population structure and particle swarm performance , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[13]  Kenneth Lange,et al.  Numerical analysis for statisticians , 1999 .

[14]  D. W. Scott,et al.  Multivariate Density Estimation, Theory, Practice and Visualization , 1992 .

[15]  T. Stützle,et al.  Iterated Local Search: Framework and Applications , 2018, Handbook of Metaheuristics.

[16]  U. Grenander.  On the theory of mortality measurement , 1956 .

[17]  B. Silverman.  Density estimation for statistics and data analysis , 1986 .

[18]  L. Birge,et al.  Estimation of unimodal densities without smoothness assumptions , 1997 .

[19]  C. D. Kemp,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[20]  L. Duembgen,et al.  Maximum likelihood estimation of a log-concave density and its distribution function: Basic properties and uniform consistency , 2007, 0709.0334.

[21]  J. Kalbfleisch.  Statistical Inference Under Order Restrictions , 1975 .

[22]  Umberto Alibrandi,et al.  Efficient evaluation of the pdf of a random variable through the kernel density maximum entropy approach , 2008 .

[23]  E. Wegman.  Nonparametric Probability Density Estimation: I. A Summary of Available Methods , 1972 .

[24]  David W. Scott,et al.  Monte Carlo Study of Three Data-Based Nonparametric Probability Density Estimators , 1981 .

[25]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[26]  Mark A. Wolters.  A Greedy Algorithm for Unimodal Kernel Density Estimation by Data Sharpening , 2012 .

[27]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[28]  Maurice Clerc,et al.  The particle swarm - explosion, stability, and convergence in a multidimensional complex space , 2002, IEEE Trans. Evol. Comput..

[29]  Daniel A. Henderson,et al.  Maximum likelihood kernel density estimation: On the potential of convolution sieves , 2009, Comput. Stat. Data Anal..

[30]  H. D. Brunk.  Maximum Likelihood Estimates of Monotone Parameters , 1955 .

[31]  M. Cule,et al.  Maximum likelihood estimation of a multi‐dimensional log‐concave density , 2008, 0804.3989.

[32]  Derek Bingham,et al.  Simulated Annealing Model Search for Subset Selection in Screening Experiments , 2011, Technometrics.

[33]  P. Hall,et al.  Data sharpening as a prelude to density estimation , 1999 .

[34]  L. Wasserman.  All of Nonparametric Statistics , 2005 .

[35]  M. C. Jones,et al.  A reliable data-based bandwidth selection method for kernel density estimation , 1991 .

[36]  Anne-Laure Fougères,et al.  Estimation de densités unimodales , 1997 .

[37]  Kee-Hoon Kang,et al.  Unimodal kernel density estimation by datra sharpening , 2005 .

[38]  Melanie Birke,et al.  Shape constrained kernel density estimation , 2009 .

[39]  Andries Petrus Engelbrecht,et al.  Fundamentals of Computational Swarm Intelligence , 2005 .

[40]  Peter Hall,et al.  UNIMODAL DENSITY ESTIMATION USING KERNEL METHODS , 2002 .

[41]  W. John Braun,et al.  Data Sharpening for Nonparametric Inference Subject to Constraints , 2001 .

[42]  Mark A. Wolters.  A particle swarm algorithm with broad applicability in shape-constrained estimation , 2012, Comput. Stat. Data Anal..

[43]  James Stephen Marron,et al.  An assessment of finite sample performance of adaptive methods in density estimation , 1999 .

[44]  Douglas G. Woolford,et al.  Convergent data sharpening for the identification and tracking of spatial temporal centers of lightning activity , 2007 .

[45]  Trevor Hastie,et al.  Non‐Parametric Logistic and Proportional Odds Regression , 1987 .

[46]  Christopher F. Parmeter,et al.  Imposing Economic Constraints in Nonparametric Regression: Survey, Implementation and Extension , 2009, SSRN Electronic Journal.

[47]  Peter J. Bickel,et al.  SOME PROBLEMS ON THE ESTIMATION OF UNIMODAL DENSITIES , 1996 .

[48]  P. Hall,et al.  NONPARAMETRIC KERNEL REGRESSION SUBJECT TO MONOTONICITY CONSTRAINTS , 2001 .

[49]  J. Ramsay.  Monotone Regression Splines in Action , 1988 .

[50]  R. Tapia,et al.  Nonparametric Probability Density Estimation , 1978 .

[51]  Jeffrey S. Racine,et al.  Nonparametric Kernel Regression with Multiple Predictors and Multiple Shape Constraints , 2012 .