Analysis of small sample size studies using nonparametric bootstrap test with pooled resampling method

Experimental studies in biomedical research frequently pose analytical problems related to small sample size. In such studies, there are conflicting findings regarding the choice of parametric and nonparametric analysis, especially with non-normal data. In such instances, some methodologists questioned the validity of parametric tests and suggested nonparametric tests. In contrast, other methodologists found nonparametric tests to be too conservative and less powerful and thus preferred using parametric tests. Some researchers have recommended using a bootstrap test; however, this method also has small sample size limitation. We used a pooled method in nonparametric bootstrap test that may overcome the problem related with small samples in hypothesis testing. The present study compared nonparametric bootstrap test with pooled resampling method corresponding to parametric, nonparametric, and permutation tests through extensive simulations under various conditions and using real data examples. The nonparametric pooled bootstrap t-test provided equal or greater power for comparing two means as compared with unpaired t-test, Welch t-test, Wilcoxon rank sum test, and permutation test while maintaining type I error probability for any conditions except for Cauchy and extreme variable lognormal distributions. In such cases, we suggest using an exact Wilcoxon rank sum test. Nonparametric bootstrap paired t-test also provided better performance than other alternatives. Nonparametric bootstrap test provided benefit over exact Kruskal-Wallis test. We suggest using nonparametric bootstrap test with pooled resampling method for comparing paired or unpaired means and for validating the one way analysis of variance test results for non-normal data in small sample size studies. Copyright © 2017 John Wiley & Sons, Ltd.

[1]  J. D. de Winter.  Using the Student ’ s t-test with extremely small sample sizes , 2013 .

[2]  Bernice W. Polemis.  Nonparametric Statistics for the Behavioral Sciences , 1959 .

[3]  G. Chase,et al.  PERMUTATION TESTS AND OTHER TEST STATISTICS FOR ILL-BEHAVED DATA: EXPERIENCE OF THE NINDS t-PA STROKE TRIAL , 2001 .

[4]  M. Gardner,et al.  Statistical guidelines for contributors to medical journals. , 1983, British medical journal.

[5]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[6]  JULIA FISCHER,et al.  Use of statistical programs for nonparametric tests of small samples often leads to incorrectPvalues: examples fromAnimal Behaviour , 1998, Animal Behaviour.

[7]  James Allan,et al.  A comparison of statistical significance tests for information retrieval evaluation , 2007, CIKM '07.

[8]  Edgar Brunner,et al.  Asymptotic permutation tests in general factorial designs , 2015 .

[9]  Siu Hui,et al.  Methods for Comparison of Cost Data , 1997, Annals of Internal Medicine.

[10]  G. Woody,et al.  Motivational interviewing to improve treatment engagement and outcome in individuals seeking treatment for substance abuse: a multisite effectiveness study. , 2006, Drug and alcohol dependence.

[11]  Amctb No,et al.  An introduction to non-parametric statistics , 2013 .

[12]  Adelchi Azzalini,et al.  The Skew-Normal and Related Families , 2018 .

[13]  C. Brownie,et al.  BOOTSTRAP P-VALUES FOR TESTS OF NONPARAMETlUC HYPOTHESES , 1988 .

[14]  Shlomo S. Sawilowsky,et al.  Comparative Power Of The Independent t, Permutation t, and WilcoxonTests , 2009 .

[15]  S. Janušonis.  Comparing two small samples with an unstable, treatment-independent baseline , 2009, Journal of Neuroscience Methods.

[16]  S. Shott,et al.  Nonparametric Statistics , 2018, The Encyclopedia of Archaeological Sciences.

[17]  Peter Hall,et al.  ON THE BOOTSTRAP AND TWO-SAMPLE PROBLEMS , 1988 .

[18]  P. Bridge,et al.  Increasing physicians' awareness of the impact of statistics on research outcomes: comparative power of the t-test and and Wilcoxon Rank-Sum test in small samples applied research. , 1999, Journal of clinical epidemiology.

[19]  R. Iman,et al.  Rank Transformations as a Bridge Between Parametric , 2007 .

[20]  Bruno D. Zumbo,et al.  Parametric Alternatives to the Student T Test under Violation of Normality and Homogeneity of Variance , 1992 .

[21]  Paul J. Fitzpatrick,et al.  Statistical Societies in the United States in the Nineteenth Century , 1957 .

[22]  Hisashi Tanizaki,et al.  Power comparison of non-parametric tests: Small-sample properties from Monte Carlo experiments , 1997 .

[23]  S. Thompson,et al.  Analysis of cost data in randomized trials: an application of the non-parametric bootstrap. , 2000, Statistics in medicine.

[24]  Frank Konietschke,et al.  Bootstrapping and permuting paired t-test type statistics , 2014, Stat. Comput..

[25]  S. Sawilowsky,et al.  Power of the independent samples t test under a prevalent psychometric measure distribution. , 1992, Journal of consulting and clinical psychology.

[26]  John Ludbrook,et al.  Why Permutation Tests are Superior to t and F Tests in Biomedical Research , 1998 .

[27]  A. Azzalini,et al.  The multivariate skew-normal distribution , 1996 .

[28]  G. Smyth,et al.  Statistical Applications in Genetics and Molecular Biology Permutation P -values Should Never Be Zero: Calculating Exact P -values When Permutations Are Randomly Drawn , 2011 .

[29]  Laura L. Lansing.  Bootstrapping versus the student's t : the problems of Type I error and power , 1999 .

[30]  Clifford E. Lunneborg,et al.  Bootstrap Applications for the Behavioral Sciences , 1987 .

[31]  A. Janssen,et al.  How do bootstrap and permutation tests work , 2003 .

[32]  D. Adams,et al.  Using randomization techniques to analyse behavioural data , 1996, Animal Behaviour.

[33]  Cyrus R. Mehta,et al.  Exact Level And Power Of Permutation, Bootstrap, And Asymptotic Tests Of Trend , 2002 .

[34]  T. Lumley,et al.  The importance of the normality assumption in large public health data sets. , 2002, Annual review of public health.

[35]  R. Sokal,et al.  Introduction to biostatistics , 1974 .

[36]  M. D. Ernst.  Permutation Methods: A Basis for Exact Inference , 2004 .

[37]  P. Good.  Permutation, Parametric, and Bootstrap Tests of Hypotheses , 2005 .

[38]  J. Jekel,et al.  Epidemiology, Biostatistics and Preventive Medicine , 1996 .

[39]  Bruce E. Wampold,et al.  Theory and Application of Statistics , 1989 .

[40]  Chooi Heng Lai,et al.  Relative power performance of t-test and bootstrap procedure for two-sample , 2012 .

[41]  J. I.  The Design of Experiments , 1936, Nature.

[42]  Arnold Janssen,et al.  A Monte Carlo comparison of studentized bootstrap and permutation tests for heteroscedastic two-sample problems , 2005, Comput. Stat..

[43]  V W Berger,et al.  Pros and cons of permutation tests in clinical trials. , 2000, Statistics in medicine.

[44]  S. Kulkarni.  A comparison of Type I error rates for the Bootstrap Contrast with the t test and the Roburst Rank Order test for various sample sizes and variances , 1993 .