Recent Developments in Nonparametric Density Estimation

Advances in computation and the fast and cheap computational facilities now available to statisticians have had a significant impact upon statistical research, and especially the development of nonparametric data analysis procedures. In particular, theoretical and applied research on nonparametric density estimation has had a noticeable influence on related topics, such as nonparametric regression, nonparametric discrimination, and nonparametric pattern recognition. This article reviews recent developments in nonparametric density estimation and includes topics that have been omitted from review articles and books on the subject. The early density estimation methods, such as the histogram, kernel estimators, and orthogonal series estimators are still very popular, and recent research on them is described. Different types of restricted maximum likelihood density estimators, including order-restricted estimators, maximum penalized likelihood estimators, and sieve estimators, are discussed, where restrictions are imposed upon the class of densities or on the form of the likelihood function. Nonparametric density estimators that are data-adaptive and lead to locally smoothed estimators are also discussed; these include variable partition histograms, estimators based on statistically equivalent blocks, nearest-neighbor estimators, variable kernel estimators, and adaptive kernel estimators. For the multivariate case, extensions of methods of univariate density estimation are usually straightforward but can be computationally expensive. A method of multivariate density estimation that did not spring from a univariate generalization is described, namely, projection pursuit density estimation, in which both dimensionality reduction and density estimation can be pursued at the same time. Finally, some areas of related research are mentioned, such as nonparametric estimation of functionals of a density, robust parametric estimation, semiparametric models, and density estimation for censored and incomplete data, directional and spherical data, and density estimation for dependent sequences of observations.

[1]  Abraham Wald,et al.  An Extension of Wilks' Method for Setting Tolerance Limits , 1943 .

[2]  J. Tukey.  Non-Parametric Estimation II. Statistically Equivalent Blocks and Tolerance Regions--The Continuous Case , 1947 .

[3]  Donald Fraser,et al.  Nonparametric Tolerance Regions , 1953 .

[4]  M. Rosenblatt.  Remarks on Some Nonparametric Estimates of a Density Function , 1956 .

[5]  U. Grenander.  On the theory of mortality measurement , 1956 .

[6]  D. Fraser.  Nonparametric methods in statistics , 1957 .

[7]  P. Whittle.  On the Smoothing of Probability Density Functions , 1958 .

[8]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[9]  M. R. Leadbetter,et al.  Hazard Analysis , 2018, System Safety Engineering and Risk Assessment.

[10]  C. Quesenberry,et al.  A nonparametric estimate of a multivariate density function , 1965 .

[11]  T. Cacoullos.  Estimation of a multivariate density , 1966 .

[12]  S. Schwartz.  Estimation of Probability Density by an Orthogonal Series , 1967 .

[13]  Tim Robertson,et al.  On Estimating a Density which is Measurable with Respect to a $\sigma$-Lattice , 1967 .

[14]  R. Kronmal,et al.  The Estimation of Probability Densities and Cumulatives by Fourier Series Methods , 1968 .

[15]  M. Gessaman,et al.  Nonparametric Discrimination Using Tolerance Regions , 1968 .

[16]  R. Bruce,et al.  Comparative Effects of Aging and Coronary Heart Disease on Submaximal and Maximal Exercise , 1969, Circulation.

[17]  Geoffrey S. Watson,et al.  Density Estimation by Orthogonal Series , 1969 .

[18]  Maximum likelihood histograms. , 1969 .

[19]  Charles T. Wolverton,et al.  Recursive Estimates of Probability Densities , 1969, IEEE Transactions on Systems Science and Cybernetics.

[20]  Ingram Olkin,et al.  Unbiased Estimation of Some Multivariate Probability Densities and Related Functions , 1969 .

[21]  M. Gessaman.  A Consistent Nonparametric Multivariate Density Estimator Based on Statistically Equivalent Blocks , 1970 .

[22]  David G. Kendall,et al.  Spline Transformations: Three New Diagnostic Aids for the Statistical Data‐Analyst , 1971 .

[23]  Grace Wahba,et al.  A Polynomial Algorithm for Density Estimation , 1971 .

[24]  H. D. Brunk,et al.  Statistical inference under order restrictions : the theory and application of isotonic regression , 1973 .

[25]  R. H. Farrell.  On the Best Obtainable Asymptotic Rates of Convergence in Estimation of a Density Function at a Point , 1972 .

[26]  M. Gessaman,et al.  A Comparison of Some Multivariate Discrimination Procedures , 1972 .

[27]  E. Wegman.  Nonparametric Probability Density Estimation: I. A Summary of Available Methods , 1972 .

[28]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[29]  J. V. Ryzin,et al.  A histogram method of density estimation , 1973 .

[30]  B. R. Crain.  A Note on Density Estimation Using Orthogonal Expansions , 1973 .

[31]  John W. Tukey,et al.  A Projection Pursuit Algorithm for Exploratory Data Analysis , 1974, IEEE Transactions on Computers.

[32]  J. Kalbfleisch.  Statistical Inference Under Order Restrictions , 1975 .

[33]  R. Tapia,et al.  Nonparametric Maximum Likelihood Estimation of Probability Densities by Penalty Function Methods , 1975 .

[34]  Edward J. Wegman,et al.  Sequential nonparametric density estimation , 1975, IEEE Trans. Inf. Theory.

[35]  T. Wagner,et al.  Nonparametric estimates of probability densities , 1975, IEEE Trans. Inf. Theory.

[36]  R. Carroll.  On sequential density estimation , 1976 .

[37]  R. Kronmal,et al.  An Introduction to the Implementation and Theory of Nonparametric Density Estimation , 1976 .

[38]  R. Kronmal,et al.  Some Classification Procedures for Multivariate Binary Data Using Orthogonal Functions , 1976 .

[39]  R. Reiss.  On minimum distance estimators for unimodal densities , 1976 .

[40]  R. Beran.  Robust Location Estimates , 1977 .

[41]  J. Yackel,et al.  Consistency Properties of Nearest Neighbor Density Function Estimators , 1977 .

[42]  L. Breiman,et al.  Variable Kernel Estimates of Multivariate Densities , 1977 .

[43]  G. Walter.  Properties of Hermite Series Estimation of Probability Density , 1977 .

[44]  H. D. Brunk.  Univariate density estimation by orthogonal series , 1978 .

[45]  J. Steele,et al.  Lower Bounds for Nonparametric Density Estimation Rates , 1978 .

[46]  W. Wertz.  Statistical density estimation: A survey , 1978 .

[47]  D. W. Scott,et al.  Plasma lipids as collateral risk factors in coronary artery disease--a study of 371 males with chest pain. , 1978, Journal of chronic diseases.

[48]  R. Tapia,et al.  Nonparametric Probability Density Estimation , 1978 .

[49]  Rui J. P. de Figueiredo,et al.  An Adaptive Orthogonal-Series Estimator for Probability Density Functions , 1978 .

[50]  B. Silverman,et al.  Density Ratios, Empirical Likelihood and Cot Death , 1978 .

[51]  Tom Leonard.  Density Estimation, Stochastic Processes and Prior Information , 1978 .

[52]  Edward J. Wegman,et al.  Remarks on Some Recursive Estimators of a Probability Density , 1979 .

[53]  G. Walter,et al.  Probability Density Estimation Using Delta Sequences , 1979 .

[54]  Hung T. Nguyen,et al.  Density Estimation in a Continuous-Time Stationary Markov Process , 1979 .

[55]  M. Rosenblatt.  Global measures of deviation for kernel and nearest neighbor density estimates , 1979 .

[56]  M. Rosenblatt,et al.  Multivariate k-nearest neighbor density estimates , 1979 .

[57]  D. W. Scott.  On optimal and data based histograms , 1979 .

[58]  David W. Scott,et al.  On Improving Convergence Rates for Nonnegative Kernel Density Estimators , 1980 .

[59]  ON DENSITY-ESTIMATION BY MEANS OF POISSONS DISTRIBUTION , 1980 .

[60]  C. Tsokos,et al.  Developments in Nonparametric Density-Estimation , 1980 .

[61]  D. W. Scott,et al.  Nonparametric Probability Density Estimation by Discrete Maximum Penalized- Likelihood Criteria , 1980 .

[62]  J. Anderson,et al.  Smooth Estimates for the Hazard Function , 1980 .

[63]  I. Good,et al.  Density Estimation and Bump-Hunting by the Penalized Likelihood Method Exemplified by Scattering and Meteorite Data , 1980 .

[64]  B. Silverman,et al.  Using Kernel Density Estimates to Investigate Multimodality , 1981 .

[65]  James R. Thompson,et al.  Some Nonparametric Techniques for Estimating the Intensity Function of a Cancer Related Nonstationary Poisson Process , 1981 .

[66]  E. F. Schuster,et al.  On the Nonconsistency of Maximum Likelihood Nonparametric Density Estimators , 1981 .

[67]  David A. Freedman,et al.  On the maximum deviation between the histogram and the underlying density , 1981 .

[68]  I. J. Good,et al.  Recent Advances in Bump Hunting , 1981 .

[69]  P. Hall.  On Trigonometric Series Estimates of Densities , 1981 .

[70]  Ian Abramson.  On Bandwidth Variation in Kernel Estimates-A Square Root Law , 1982 .

[71]  S. Geman,et al.  Nonparametric Maximum Likelihood Estimation by the Method of Sieves , 1982 .

[72]  V. Klonias.  Consistency of Two Nonparametric Maximum Penalized Likelihood Estimators of the Probability Density Function , 1982 .

[73]  Jerome H. Friedman,et al.  Projection Pursuit Methods for Data Analysis. , 1982 .

[74]  T. Sager.  Nonparametric Maximum Likelihood Estimation of Spatial Patterns , 1982 .

[75]  Josef Kittler,et al.  Pattern recognition : a statistical approach , 1982 .

[76]  David J. Hand,et al.  Kernel Discriminant Analysis , 1983 .

[77]  Peter Hall,et al.  Cross-validation in density estimation , 1982 .

[78]  Prakasa Rao.  Nonparametric functional estimation , 1983 .

[79]  Nozer D. Singpurwalla,et al.  Estimation of the failure rate-a survey of nonparametric methods Part I: Non-Bayesian Methods , 1983 .

[80]  Brian S. Yandell,et al.  Nonparametric Inference for Rates with Censored Survival Data , 1983 .

[81]  B. W. Silverman,et al.  Probability, Statistics and Analysis: Some properties of a test for multimodality based on kernel density estimates , 1983 .

[82]  Martin A. Tanner,et al.  The Estimation of the Hazard Function from Randomly Censored Data by the Kernel Method , 1983 .

[83]  S. Geman,et al.  Consistent Cross-Validated Density Estimation , 1983 .

[84]  Martin A. Tanner,et al.  A Note on the Variable Kernel Estimator of the Hazard Function from Randomly Censored Data , 1983 .

[85]  D. M. Titterington,et al.  Kernel‐Based Density Estimates from Incomplete Data , 1983 .

[86]  P. Groeneboom.  Estimating a monotone density , 1984 .

[87]  J. Friedman,et al.  PROJECTION PURSUIT DENSITY ESTIMATION , 1984 .

[88]  W. J. Padgett,et al.  Nonparametric density estimation from censored data , 1984 .

[89]  Jeffrey D. Hart,et al.  Efficiency of a Kernel Density Estimator under an Autoregressive Dependence Model , 1984 .

[90]  B. Silverman,et al.  Spline Smoothing: The Equivalent Variable Kernel Method , 1984 .

[91]  V. K. Klonias.  On a Class of Nonparametric Density and Regression Estimators , 1984 .

[92]  M. Falk.  Relative Deficiency of Kernel Type Estimators of Quantiles , 1984 .

[93]  A. Bowman.  An alternative method of cross-validation for the smoothing of density estimates , 1984 .

[94]  M. C. Jones,et al.  A Remark on Algorithm as 176. Kernel Density Estimation Using the Fast Fourier Transform , 1984 .

[95]  C. J. Stone,et al.  An Asymptotically Optimal Window Selection Rule for Kernel Density Estimates , 1984 .

[96]  Luc Devroye,et al.  The Consistency of Automatic Kernel Density Estimates , 1984 .

[97]  R. Kronmal,et al.  Discrimination with Polychotomous Predictor Variables Using Orthogonal Functions , 1985 .

[98]  J. Hart.  On the choice of a truncation point in fourier series density estimation , 1985 .

[99]  J. Aitchison,et al.  Kernel Density Estimation for Compositional Data , 1985 .

[100]  A Histogram Estimator of the Hazard Rate with Censored Data , 1985 .

[101]  A Note on Data-Adaptive Kernel Estimation of the Hazard and Density Function in the Random Censorship Situation , 1985 .

[102]  H. Müller,et al.  Kernels for Nonparametric Curve Estimation , 1985 .

[103]  L. Devroye,et al.  Nonparametric Density Estimation: The L 1 View. , 1985 .

[104]  D. W. Scott.  Averaged Shifted Histograms: Effective Nonparametric Density Estimators in Several Dimensions , 1985 .

[105]  C. C. Rodriguez,et al.  Maximum entropy histograms , 1985 .

[106]  Shie-Shien Yang.  A Smooth Nonparametric Estimator of a Quantile Function , 1985 .

[107]  J. Marron.  An Asymptotically Efficient Solution to the Bandwidth Problem of Kernel Density Estimation , 1985 .

[108]  Nonparametric maximum penalized likelihood estimation of a density from arbitrarily right-censored observations , 1985 .

[109]  B. Silverman.  Two Books on Density Estimation , 1985 .

[110]  Lesław Gajek,et al.  On Improving Density Estimators which are not Bona Fide Functions , 1986 .

[111]  L. Denby,et al.  The survival curve with decreasing density , 1986 .

[112]  F. O’Sullivan.  A Statistical Perspective on Ill-posed Inverse Problems , 1986 .

[113]  Elias Masry,et al.  Recursive probability density estimation for weakly dependent stationary processes , 1986, IEEE Trans. Inf. Theory.

[114]  An Application of Isotonic Regression to Multivariate Density Estimation , 1986 .

[115]  James Stephen Marron,et al.  Random approximations to some measures of accuracy in nonparametric curve estimation , 1986 .

[116]  R. Tamura,et al.  Minimum Hellinger Distance Estimation for Multivariate Location and Covariance , 1986 .

[117]  Kernel density estimations applied to gamma ray light curves , 1986 .

[118]  Robin Sibson,et al.  What is projection pursuit , 1987 .

[119]  P. Hall,et al.  Kernel density estimation with spherical data , 1987 .

[120]  D. G. Simpson,et al.  Minimum Hellinger Distance Estimation for the Analysis of Count Data , 1987 .

[121]  Atsuyuki Kogure,et al.  Asymptotically Optimal Cells for a Historgram , 1987 .

[122]  J. Marron,et al.  Extent to which least-squares cross-validation minimises integrated square error in nonparametric density estimation , 1987 .

[123]  L. Devroye.  A Course in Density Estimation , 1987 .

[124]  James Stephen Marron,et al.  Asymptotically Optimal Bandwidth Selection for Kernel Density Estimators from Randomly Right-Censored Samples. , 1987 .

[125]  D. W. Scott,et al.  Biased and Unbiased Cross-Validation in Density Estimation , 1987 .

[126]  G. A. Young,et al.  The bootstrap: To smooth or not to smooth? , 1987 .

[127]  J. Friedman.  Exploratory Projection Pursuit , 1987 .

[128]  Ingram Olkin,et al.  A Semiparametric Approach to Density Estimation , 1987 .

[129]  L. Györfi,et al.  Strong consistency and rates for recursive probability density estimators of stationary processes , 1987 .

[130]  E. Hannan,et al.  On stochastic complexity and nonparametric density estimation , 1988 .

[131]  D. Donoho.  One-sided inference about functionals of a density , 1988 .

[132]  A. Izenman,et al.  Philatelic Mixtures and Multimodal Densities , 1988 .

[133]  James Stephen Marron,et al.  Comparison of data-driven bandwith selectors , 1988 .

[134]  F. T. Wright,et al.  Order restricted statistical inference , 1988 .

[135]  Daren B. H. Cline,et al.  Admissibile Kernel Estimators of a Multivariate Density , 1988 .

[136]  Matt P. Wand,et al.  Minimizing L 1 distance in nonparametric density estimation , 1988 .

[137]  Yuichiro Kanazawa.  An optimal variable cell histogram , 1988 .

[138]  D. W. Scott,et al.  Nonparametric Estimation of Probability Densities and Regression Curves , 1988 .

[139]  James Stephen Marron,et al.  Choice of Kernel Order in Density Estimation , 1988 .

[140]  Calyampudi R. Rao,et al.  Kernel estimators of density function of directional data , 1988 .

[141]  James Stephen Marron,et al.  Canonical kernels for density estimation , 1988 .

[142]  H. Joe.  Estimation of entropy and other functionals of a multivariate density , 1989 .

[143]  D. G. Simpson,et al.  Hellinger Deviance Tests: Efficiency, Breakdown Points, and Examples , 1989 .

[144]  Thomas J. DiCiccio,et al.  On Smoothing and the Bootstrap , 1989 .

[145]  M. C. Jones,et al.  Spline Smoothing and Nonparametric Regression. , 1989 .

[146]  Luc Devroye,et al.  On the Relationship Between Stability of Extreme Order Statistics and Convergence of the Maximum Likelihood Kernel Density Estimate , 1989 .

[147]  M. C. Jones.  Discretized and Interpolated Kernel Density Estimates , 1989 .

[148]  G. Roussas.  Asymptotic normality of the kernel estimate under dependence conditions: application to hazard rate , 1990 .

[149]  James Stephen Marron,et al.  Kernel Quantile Estimators , 1990 .

[150]  G. Terrell.  The Maximal Smoothing Principle in Density Estimation , 1990 .

[151]  Peter Hall,et al.  Using the bootstrap to estimate mean squared error and select smoothing parameter in nonparametric problems , 1990 .

[152]  A. Izenman,et al.  Kernel estimation of the survival function and hazard rate under weak dependence , 1990 .

[153]  Harrie Hendriks,et al.  Nonparametric Estimation of a Probability Density on a Riemannian Manifold Using Fourier Expansions , 1990 .

[154]  Bernard W. Silverman,et al.  Speed of Estimation in Positron Emission Tomography and Related Inverse Problems , 1990 .

[155]  A. Izenman.  Review Papers: Recent Developments in Nonparametric Density Estimation , 1991 .