Order statistics and nearest neighbors

We start with some basic properties of uniform order statistics. For a general introduction to probability, see Grimmett and Stirzaker (2001). Some of the properties of order statistics presented in this chapter are covered by Renyi (1970); Galambos (1978), and Devroye (1986).

[1]  A. C. Berry.  The accuracy of the Gaussian approximation to the sum of independent variates , 1941 .

[2]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[3]  J. L. Hodges,et al.  Discriminatory Analysis - Nonparametric Discrimination: Small Sample Performance , 1952 .

[4]  H. Chernoff.  A Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations , 1952 .

[5]  H. Akaike.  An approximation to the density function , 1954 .

[6]  M. Rosenblatt.  Remarks on Some Nonparametric Estimates of a Density Function , 1956 .

[7]  P. Whittle.  On the Smoothing of Probability Density Functions , 1958 .

[8]  A. Rényi.  On Measures of Entropy and Information , 1961 .

[9]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[10]  G. Bennett.  Probability Inequalities for the Sum of Independent Random Variables , 1962 .

[11]  M. R. Leadbetter,et al.  On the Estimation of the Probability Density, I , 1963 .

[12]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[13]  E. Nadaraya.  On Estimating Regression , 1964 .

[14]  G. S. Watson,et al.  Smooth regression analysis , 1964 .

[15]  H. O. Posten.  Multidimensional Gaussian Distributions , 1964 .

[16]  J. Milnor.  On the Betti numbers of real varieties , 1964 .

[17]  C. Quesenberry,et al.  A nonparametric estimate of a multivariate density function , 1965 .

[18]  E. Nadaraya.  On Non-Parametric Estimates of Density Functions and Regression Curves , 1965 .

[19]  T. Cacoullos.  Estimation of a multivariate density , 1966 .

[20]  L. Turner,et al.  Inverse of the Vandermonde matrix with applications , 1966 .

[21]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[22]  H. Warren.  Lower bounds for approximation by nonlinear manifolds , 1968 .

[23]  Thomas M. Cover,et al.  Estimation by the nearest neighbor rule , 1968, IEEE Trans. Inf. Theory.

[24]  T. Wagner,et al.  Asymptotically optimal discriminant functions for pattern classification , 1969, IEEE Trans. Inf. Theory.

[25]  E. Henrichon,et al.  Uniform Consistency of Some Estimates of a Density Function , 1969 .

[26]  E. Stein.  Singular Integrals and Di?erentiability Properties of Functions , 1971 .

[27]  Charles Fefferman,et al.  Some Maximal Inequalities , 1971 .

[28]  Jovan D. Kečkić,et al.  Some Inequalities For The Gamma Function , 1971 .

[29]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[30]  B. Grünbaum.  Arrangements and Spreads , 1972 .

[31]  N. Glick.  Sample-Based Multinomial Classification , 1973 .

[32]  József Fritz,et al.  Distribution-free exponential error bound for nearest neighbor pattern classification , 1975, IEEE Trans. Inf. Theory.

[33]  V. V. Petrov.  Sums of Independent Random Variables , 1975 .

[34]  J. V. Ryzin,et al.  Uniform consistency of a histogram density estimator and modal estimation , 1975 .

[35]  C. J. Stone,et al.  Consistent Nonparametric Regression , 1977 .

[36]  A. Zygmund,et al.  Measure and integral : an introduction to real analysis , 1977 .

[37]  Jan M. Van Campenhout,et al.  On the Possible Orderings in the Measurement Selection Problem , 1977, IEEE Transactions on Systems, Man, and Cybernetics.

[38]  J. Yackel,et al.  Consistency Properties of Nearest Neighbor Density Function Estimators , 1977 .

[39]  L. Devroye,et al.  The Strong Uniform Consistency of Nearest Neighbor Density Estimates. , 1977 .

[40]  J. Yackel,et al.  Large Sample Properties of Nearest Neighbor Density Function Estimators , 1977 .

[41]  J. D. T. Oliveira,et al.  The Asymptotic Theory of Extreme Order Statistics , 1979 .

[42]  L. Breiman,et al.  Variable Kernel Estimates of Multivariate Densities , 1977 .

[43]  Pierre A. Devijver.  A note on ties in voting with the k-NN rule , 1978, Pattern Recognit..

[44]  Anil K. Jain,et al.  NOTE ON DISTANCE-WEIGHTED k-NEAREST NEIGHBOR RULES. , 1978 .

[45]  László Györfi,et al.  An upper bound on the asymptotic error probability on the k-nearest neighbor rule for multiple classes (Corresp.) , 1978, IEEE Trans. Inf. Theory.

[46]  L. Gyorfi.  On the rate of convergence of nearest neighbor rules (Corresp.) , 1978 .

[47]  Luc Devroye,et al.  The uniform convergence of nearest neighbor regression function estimators and their application in optimization , 1978, IEEE Trans. Inf. Theory.

[48]  Pierre A. Devijver,et al.  New error bounds with the nearest neighbor rule (Corresp.) , 1979, IEEE Trans. Inf. Theory.

[49]  M. Rosenblatt,et al.  Multivariate k-nearest neighbor density estimates , 1979 .

[50]  G. Collomb.  Estimation de la regression par la methode des k points les plus proches avec noyau : quelques propriétés de convergence ponctuelle , 1980 .

[51]  C. J. Stone,et al.  Optimal Rates of Convergence for Nonparametric Estimators , 1980 .

[52]  L. Devroye.  On the Asymptotic Probability of Error in Nonparametric Discrimination , 1981 .

[53]  G. Collomb.  Estimation Non-paramétrique de la Régression: Revue Bibliographique@@@Estimation Non-parametrique de la Regression: Revue Bibliographique , 1981 .

[54]  Y. Mack,et al.  Local Properties of k-NN Regression Estimates , 1981 .

[55]  Luc Devroye,et al.  On the Inequality of Cover and Hart in Nearest Neighbor Discrimination , 1981, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  B. Efron,et al.  The Jackknife Estimate of Variance , 1981 .

[57]  L. Devroye.  On the Almost Everywhere Convergence of Nonparametric Regression Function Estimates , 1981 .

[58]  C. J. Stone,et al.  Optimal Global Rates of Convergence for Nonparametric Regression , 1982 .

[59]  L. Devroye,et al.  8 Nearest neighbor methods in discrimination , 1982, Classification, Pattern Recognition and Reduction of Dimensionality.

[60]  G. Grimmett,et al.  Probability and random processes , 2002 .

[61]  L. Devroye.  Necessary and sufficient conditions for the pointwise convergence of nearest neighbor regression function estimates , 1982 .

[62]  Prakasa Rao.  Nonparametric functional estimation , 1983 .

[63]  Peter Hall,et al.  On near neighbour estimates of a multivariate density , 1983 .

[64]  Y. Mack,et al.  Rate of strong uniform convergence of k-NN density estimates , 1983 .

[65]  P. Bickel,et al.  Sums of Functions of Nearest Neighbor Distances, Moment Bounds, Limit Theorems and a Goodness of Fit Test , 1983 .

[66]  W. Stute.  Asymptotic Normality of Nearest Neighbor Regression Function Estimates , 1984 .

[67]  P. Cheng.  Strong consistency of nearest neighbor regression function estimators , 1984 .

[68]  Luc Devroye,et al.  Nonparametric Density Estimation , 1985 .

[69]  C. C. Rodriguez,et al.  Maximum entropy histograms , 1985 .

[70]  J. Kiefer.  Iterated Logarithm Analogues for Sample Quantiles When P n ↓0 , 1985 .

[71]  B. Silverman.  Density estimation for statistics and data analysis , 1986 .

[72]  L. Devroye.  Non-Uniform Random Variate Generation , 1986 .

[73]  John Van Ryzin,et al.  Large sample properties of maximum entropy histograms , 1986, IEEE Trans. Inf. Theory.

[74]  J. Steele.  An Efron-Stein inequality for nonsymmetric statistics , 1986 .

[75]  L. Zhao.  Exponential bounds of mean error for the nearest neighbor estimates of regression functions*1 , 1987 .

[76]  Micha Hofri,et al.  Probabilistic Analysis of Algorithms , 1987, Texts and Monographs in Computer Science.

[77]  D. Donoho.  One-sided inference about functionals of a density , 1988 .

[78]  Luc Devroye,et al.  Automatic Pattern Recognition: A Study of the Probability of Error , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[79]  J. L. Hodges,et al.  Discriminatory Analysis - Nonparametric Discrimination: Consistency Properties , 1989 .

[80]  Colin McDiarmid,et al.  Surveys in Combinatorics, 1989: On the method of bounded differences , 1989 .

[81]  Torben Hagerup,et al.  A Guided Tour of Chernoff Bounds , 1990, Inf. Process. Lett..

[82]  Belur V. Dasarathy,et al.  Nearest neighbor (NN) norms: NN pattern classification techniques , 1991 .

[83]  L. Devroye.  Exponential Inequalities in Nonparametric Estimation , 1991 .

[84]  David W. Scott,et al.  Multivariate Density Estimation: Theory, Practice, and Visualization , 1992, Wiley Series in Probability and Statistics.

[85]  Rolf-Dieter Reiss,et al.  On conditional distributions of nearest neighbors , 1992 .

[86]  D. W. Scott,et al.  Multivariate Density Estimation, Theory, Practice and Visualization , 1992 .

[87]  R. Pollack,et al.  On the number of cells defined by a set of polynomials , 1993 .

[88]  A. Tsybakov,et al.  Root-N consistent estimators of entropy for densities with unbounded support , 1994, Proceedings of 1994 Workshop on Information Theory and Statistics.

[89]  G. Lugosi,et al.  On the Strong Universal Consistency of Nearest Neighbor Regression Function Estimates , 1994 .

[90]  Sanjeev R. Kulkarni,et al.  Rates of convergence of nearest neighbor estimation under arbitrary sampling , 1995, IEEE Trans. Inf. Theory.

[91]  P. Massart,et al.  Estimation of Integral Functionals of a Density , 1995 .

[92]  B. Laurent.  Efficient estimation of integral functionals of a density , 1996 .

[93]  László Györfi,et al.  A Probabilistic Theory of Pattern Recognition , 1996, Stochastic Modelling and Applied Probability.

[94]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[95]  L. Györfi,et al.  Nonparametric entropy estimation. An overview , 1997 .

[96]  S. Rachev,et al.  Mass transportation problems , 1998 .

[97]  A. V. D. Vaart,et al.  Asymptotic Statistics: Frontmatter , 1998 .

[98]  László Györfi,et al.  Lower Bounds for Bayes Error Estimation , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[99]  Larry George.  Mathematical Statistics, a Unified Introduction , 2000, Technometrics.

[100]  A. Berlinet,et al.  Higher Order Analysis at Lebesgue Points , 2000 .

[101]  Luc Devroye,et al.  Combinatorial methods in density estimation , 2001, Springer series in statistics.

[102]  C. C. Rodriguez.  Optimal recovery of local truth , 2000, physics/0010063.

[103]  Adam Krzyzak,et al.  New Multivariate Product Density Estimators , 2002 .

[104]  Antonia J. Jones,et al.  Asymptotic moments of near–neighbour distance distributions , 2002, Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[105]  Jiri Matousek,et al.  Lectures on discrete geometry , 2002, Graduate texts in mathematics.

[106]  Adam Krzyzak,et al.  A Distribution-Free Theory of Nonparametric Regression , 2002, Springer series in statistics.

[107]  A. Tsybakov,et al.  Optimal aggregation of classifiers in statistical learning , 2003 .

[108]  K. Böröczky,et al.  Covering the Sphere by Equal Spherical Balls , 2003 .

[109]  Györfi László,et al.  The estimation problem of minimum mean squared error , 2003 .

[110]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[111]  K. Böröczky.  Finite Packing and Covering , 2004 .

[112]  M. Giaquinta,et al.  Mathematical Analysis: An Introduction to Functions of Several Variables , 2004 .

[113]  Adam Krzyżak,et al.  Rates of convergence for partitioning and nearest neighbor regression estimates with unbounded data , 2006 .

[114]  Arnaud Guyader,et al.  Nearest neighbor classification in infinite dimension , 2006 .

[115]  Adam Krzyzak,et al.  On the Rate of Convergence of Local Averaging Plug-In Classification Rules Under a Margin Condition , 2007, IEEE Transactions on Information Theory.

[116]  P. Massart,et al.  Concentration inequalities and model selection , 2007 .

[117]  P. Massart,et al.  Risk bounds for statistical learning , 2007, math/0702683.

[118]  A. Tsybakov,et al.  Fast learning rates for plug-in classifiers , 2007, 0708.2321.

[119]  Amaury Lendasse,et al.  Non-parametric Residual Variance Estimation in Supervised Learning , 2007, IWANN.

[120]  Harro Walk,et al.  A universal strong law of large numbers for conditional expectations via nearest neighbors , 2008 .

[121]  Amaury Lendasse,et al.  On Nonparametric Residual Variance Estimation , 2008, Neural Processing Letters.

[122]  L. Pronzato,et al.  A class of Rényi information estimators for multidimensional densities , 2008, 0810.5302.

[123]  A. Lendasse,et al.  Bounds on the mean power-weighted nearest neighbour distance , 2008, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[124]  Alexandre B. Tsybakov,et al.  Introduction to Nonparametric Estimation , 2008, Springer series in statistics.

[125]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[126]  Amaury Lendasse,et al.  Residual variance estimation using a nearest neighbor statistic , 2010, J. Multivar. Anal..

[127]  Arnaud Guyader,et al.  On the Rate of Convergence of the Bagged Nearest Neighbor Estimate , 2010, J. Mach. Learn. Res..

[128]  J. Yukich,et al.  Laws of Large Numbers and Nearest Neighbor Distances , 2009, 0911.0331.

[129]  L. Devroye,et al.  A weighted k-nearest neighbor density estimate for geometric inference , 2011 .

[130]  Arnaud Guyader,et al.  New insights into Approximate Bayesian Computation , 2012, 1207.6461.

[131]  Adam Krzyzak,et al.  An affine invariant k-nearest neighbor regression estimate , 2012, J. Multivar. Anal..

[132]  R. Samworth.  Optimal weighted nearest neighbour classifiers , 2011, 1101.5783.

[133]  Gábor Lugosi,et al.  Concentration Inequalities - A Nonasymptotic Theory of Independence , 2013, Concentration Inequalities.

[134]  T. Klein,et al.  Classification with the nearest neighbor rule in general finite dimensional spaces , 2014, 1411.0894.

[135]  Erran L. Li,et al.  LTE radio analytics made easy and accessible , 2014 .

[136]  Vipin Kumar,et al.  Feature Selection: A literature Review , 2014, Smart Comput. Rev..