Robust directed tests of normality against heavy-tailed alternatives

Many statistical procedures rely on the assumption that the observed data are normally distributed. Consequently, there exists a vast literature on tests of normality and their statistical properties. Today the most commonly used omnibus test for general use is the Shapiro-Wilk method while the Jarque-Bera test is the most popular omnibus test in economics and related applications. In a recent securities law case concerning an allegation of profit sharing, an inference is drawn on a relationship between the hypothetical profits, which customers made on days with special stock offerings, and the ratio of their commission business to those profits became an issue. Two common statistical tests are based on the Pearson or Spearman correlation coefficients. The results provided by the two tests are dramatically different with contradictory implications for the existence of a ''profit sharing arrangement''. In particular, the Pearson correlation coefficient is of -0.13 with a p-value of 0.35 while the Spearman coefficient is of -0.68 with a p-value less than 0.0001. However, if the data do not come from a bivariate normal distribution, the usual sampling theory for the Pearson correlation would not be applicable. As stock market data are typically heavy-tailed, a simple test of normality directed against heavy-tailed alternatives complemented by a graphical method is desired for presentation in a courtroom. In this paper we utilize the idea of the Shapiro-Wilk test whose test statistic W is the ratio of the classical standard deviation s"n to the optimal L-estimator of the population spread. In particular, we replace the mean and standard deviation by the median and a robust estimator of spread in the Shapiro-Wilk statistic W. This eventually leads to a new more powerful directed test and a clearer QQ plot for assessing departures from normality in the tails. Somewhat surprisingly, the robust estimator of spread that yields the more powerful test of normality utilizes the average absolute deviation from the median (MAAD) while the statistic based on the median absolute deviation from the median (MAD) usually provides a clearer graphical display. The asymptotic distribution of the proposed test statistic is derived, and the use of the new test is illustrated by simulations and by application to data sets from the securities law, finance and meteorology.

[1]  D. Bonett,et al.  A test of normality with high uniform power , 2002 .

[2]  D. F. Andrews,et al.  Robust Estimates of Location , 1972 .

[3]  Joseph L. Gastwirth,et al.  Statistical properties of a measure of tax assessment uniformity , 1982 .

[4]  P. Rodrigues,et al.  Threshold Cointegration and the PPP Hypothesis , 2004 .

[5]  P. J. Huber.  The 1972 Wald Lecture Robust Statistics: A Review , 1972 .

[6]  Weiwen Miao,et al.  A New Test of Symmetry about an Unknown Median , 2006 .

[7]  M. Hubert,et al.  A Robust Measure of Skewness , 2004 .

[8]  J. Royston.  An Extension of Shapiro and Wilk's W Test for Normality to Large Samples , 1982 .

[9]  Weiwen Miao,et al.  The Importance of Checking the Assumptions Underlying Statistical Analysis: Graphical Methods for Assessing Normality , 2005 .

[10]  Anil K. Bera,et al.  Efficient tests for normality, homoscedasticity and serial independence of regression residuals , 1980 .

[11]  R. Geary.  MOMENTS OF THE RATIO OF THE MEAN DEVIATION TO THE STANDARD DEVIATION FOR NORMAL SAMPLES , 1936 .

[12]  Joseph L. Gastwirth,et al.  Statistical Methods for Assessing the Fairness of the Allocation of Shares in Initial Public Offerings , 2005 .

[13]  Tilmann Gneiting,et al.  Calibrated Probabilistic Mesoscale Weather Field Forecasting , 2004 .

[14]  K. Sarkadi,et al.  The Asymptotic Distribution of Certain Goodness of Fit Test Statistics , 1981 .

[15]  Joseph L. Gastwirth,et al.  Asymptotic Distribution of Linear Combinations of Functions of Order Statistics with Applications to Estimation , 1967 .

[16]  J. Gastwirth.  ON ROBUST PROCEDURES , 1966 .

[17]  T. A. Bancroft,et al.  ANALYSIS AND INFERENCE FOR INCOMPLETELY SPECIFIED MODELS INVOLVING THE USE OF PRELIMINARY TEST(S) OF SIGNIFICANCE , 1964 .

[18]  H. Büning,et al.  Jarque–Bera Test and its Competitors for Testing Normality – A Power Comparison , 2007 .

[19]  Joseph L. Gastwirth,et al.  The use of the Lorenz curve, Gini index and related measures of relative inequality and uniformity in securities law , 2005 .

[20]  W. R. Buckland,et al.  Contributions to Probability and Statistics , 1960 .

[21]  R. Serfling.  Approximation Theorems of Mathematical Statistics , 1980 .

[22]  David E. Booth,et al.  Multivariate statistical inference and applications , 1997 .

[23]  S. Shapiro,et al.  An Analysis of Variance Test for Normality (Complete Samples) , 1965 .

[24]  Allan Birnbaum,et al.  Optimal Robustness: A General Method, with Applications to Linear Estimators of Location , 1967 .

[25]  Zhiliang Ying,et al.  Random walk, sequential analysis and related topics , 2013 .

[26]  H. Lilliefors.  On the Kolmogorov-Smirnov Test for Normality with Mean and Variance Unknown , 1967 .

[27]  J. Leslie,et al.  Asymptotic Distribution of the Shapiro-Wilk $W$ for Testing for Normality , 1986 .

[28]  M. M. Siddiqui,et al.  Robust Estimation of Location , 1967 .

[29]  Weiwen Miao,et al.  On the Use of the Shapiro‐Wilk Test in Two‐Stage Adaptive Inference for Paired Data from Moderate to Very Heavy Tailed Distributions , 2003 .

[30]  P. Hall,et al.  Limit theorems for the median deviation , 1985 .

[31]  Ralph B. D'Agostino,et al.  Goodness-of-Fit-Techniques , 2020 .