On the exact Berk-Jones statistics and their p-value calculation

Continuous goodness-of-fit testing is a classical problem in statistics. Despite having low power for detecting deviations at the tail of a distribution, the most popular test is based on the Kolmogorov-Smirnov statistic. While similar variance-weighted statistics, such as Anderson-Darling and the Higher Criticism statistic give more weight to tail deviations, as shown in various works, they still mishandle the extreme tails. As a viable alternative, in this paper we study some of the statistical properties of the exact $M_n$ statistics of Berk and Jones. We derive the asymptotic null distributions of $M_n, M_n^+, M_n^-$, and further prove their consistency as well as asymptotic optimality for a wide range of rare-weak mixture models. Additionally, we present a new computationally efficient method to calculate $p$-values for any supremum-based one-sided statistic, including the one-sided $M_n^+,M_n^-$ and $R_n^+,R_n^-$ statistics of Berk and Jones and the Higher Criticism statistic. We illustrate our theoretical analysis with several finite-sample simulations.

[1]  Boaz Nadler,et al.  Fast calculation of boundary crossing probabilities for Poisson processes , 2015, 1503.04363.

[2]  H. Finner,et al.  The intermediates take it all: Asymptotics of higher criticism statistics and a powerful alternative based on equal local levels , 2015, Biometrical journal. Biometrische Zeitschrift.

[3]  Jian Li,et al.  Higher criticism: $p$-values and criticism , 2014, 1411.1437.

[4]  Xihong Lin,et al.  Analytic P-value calculation for the higher criticism test in finite d problems. , 2014, Biometrika.

[5]  David Mary,et al.  A non-asymptotic standardization of binomial counts in Higher Criticism , 2014, 2014 IEEE International Symposium on Information Theory.

[6]  Yihong Wu,et al.  Optimal Detection of Sparse Mixtures Against a Given Null Distribution , 2014, IEEE Transactions on Information Theory.

[7]  J. Wellner,et al.  Confidence Bands for Distribution Functions: A New Look at the Law of the Iterated Logarithm , 2014 .

[8]  David M. Kaplan,et al.  True equality (of pointwise sensitivity) at last: a Dirichlet alternative to Kolmogorov{Smirnov inference on distributions , 2014 .

[9]  Robert A. Stine,et al.  The Power to See: A New Graphical Test of Normality , 2013 .

[10]  Guenther Walther,et al.  The average likelihood ratio for large-scale multiple testing and detecting sparse mixtures , 2011, 1111.0328.

[11]  Yihong Wu,et al.  Optimal Detection For Sparse Mixtures , 2012, ArXiv.

[12]  Olivier Thas,et al.  Smooth Tests of Goodness of Fit: Using R , 2009 .

[13]  D. Donoho,et al.  Higher criticism thresholding: Optimal feature selection when useful features are rare and weak , 2008, Proceedings of the National Academy of Sciences.

[14]  J. R. Brown,et al.  Arbitrary Precision Mathematica Functions to Evaluate the One-Sided One Sample K-S Cumulative Sampling Distribution , 2008 .

[15]  J. Randall Brown,et al.  Rational Arithmetic Mathematica Functions to Evaluate the One-sided One-sample K-S Cumulative Sample Distribution , 2007 .

[16]  J. Wellner,et al.  GOODNESS-OF-FIT TESTS VIA PHI-DIVERGENCES , 2006, math/0603238.

[17]  D. Donoho,et al.  Higher criticism for detecting sparse heterogeneous mixtures , 2004, math/0410072.

[18]  G. Marsaglia,et al.  Evaluating Kolmogorov's distribution , 2003 .

[19]  E. Khmaladze,et al.  Calculation of noncrossing probabilities for Poisson processes and its corollaries , 2001, Advances in Applied Probability.

[20]  A. Janssen.  GLOBAL POWER FUNCTIONS OF GOODNESS OF FIT TESTS , 2000 .

[21]  Thomas Friedrich,et al.  Computation of the percentage points and the power for the two-sided Kolmogorov-Smirnov one sample test , 1998 .

[22]  Art B. Owen,et al.  Nonparametric Likelihood Confidence Bands for a Distribution Function , 1995 .

[23]  T. Ledwina.  Data-Driven Version of Neyman's Smooth Test of Fit , 1994 .

[24]  John C. W. Rayner,et al.  Smooth test of goodness of fit , 1989 .

[25]  F. Calitz.  An alternative to the Kolmogorov-Smirnov test for goodness of fit , 1987 .

[26]  D. Mason,et al.  Weighted Empirical and Quantile Processes , 1986 .

[27]  A decomposition of the beta distribution, related order and asymptotic behavior , 1983 .

[28]  J. H. Schuenemeyer,et al.  A Modified Kolmogorov-Smirnov Test Sensitive to Tail Alternatives , 1983 .

[29]  S. Karlin,et al.  Classes of orderings of measures and related correlation inequalities. I. Multivariate totally positive distributions , 1980 .

[30]  Douglas H. Jones,et al.  Goodness-of-fit test statistics that dominate the Kolmogorov statistics , 1979 .

[31]  D. Jaeschke.  The Asymptotic Distribution of the Supremum of the Standardized Empirical Distribution Function on Subintervals , 1979 .

[32]  F. Eicker.  The Asymptotic Distribution of the Suprema of the Standardized Empirical Processes , 1979 .

[33]  Douglas H. Jones,et al.  Relatively Optimal Combinations of Test Statistics , 1978 .

[34]  J. Annas.  On the ”Intermediates“ , 1975 .

[35]  Geoffrey S. Watson,et al.  Distribution Theory for Tests Based on the Sample Distribution Function , 1973 .

[36]  Marc Noe,et al.  The Calculation of Distributions of Two-Sided Kolmogorov-Smirnov Type Statistics , 1972 .

[37]  John W. Pratt,et al.  A Normal Approximation for Binomial, F, Beta, and other Common, Related Tail Probabilities, II , 1968 .

[38]  D. Darling,et al.  A Test of Goodness of Fit , 1954 .

[39]  T. W. Anderson,et al.  Asymptotic Theory of Certain "Goodness of Fit" Criteria Based on Stochastic Processes , 1952 .

[40]  H. Daniels.  The statistical theory of the strength of bundles of threads. I , 1945, Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences.

[41]  J. Neyman.  »Smooth test» for goodness of fit , 1937 .