Enhancing Power While Controlling Family-Wise Error: An Illustration of the Issues Using Electrocortical Studies

This study examined the relative family-wise error (FWE) rate and statistical power of multivariate permutation tests (MPTs), Bonferroni-adjusted alpha, and uncorrected-alpha tests of significance for bivariate associations. Although there are many previous applications of MPTs, this is the first to apply it to testing bivariate associations. Electrocortical studies were selected as an example class because the sample sizes that are typical of electrocortical studies published in 2001 and 2002 are small and their multiple significance tests are typically nonindependent. Because Bonferroni adjustments assume independent predictors, we expected that MPTs would be more powerful than the Bonferroni adjustment. Results support the following conclusions: (a) failure to control for multiple significance testing results in unacceptable FWE rates, (b) the FWE rate for the MPTs approximated the alpha set for the analyses, and (c) the statistical power advantage that MPTs provide over Bonferroni adjustments is important when using small sample sizes such as those that are typical of recent electrocortical studies.

[1]  R. Thatcher.  Functional neuroimaging : technical foundations , 1994 .

[2]  A. Greenwald.  Consequences of Prejudice Against the Null Hypothesis , 1975 .

[3]  A. Tamhane,et al.  Multiple Comparison Procedures , 1989 .

[4]  G. Dawson,et al.  A comparison of hemispheric asymmetries in speech-related brain potentials of autistic and dysphasic children , 1989, Brain and Language.

[5]  Darryl G. Humphrey,et al.  Assessment of mental workload with task-irrelevant auditory probes , 1995, Biological Psychology.

[6]  J. Booth,et al.  Resampling-Based Multiple Testing. , 1994 .

[7]  J D Watson,et al.  Nonparametric Analysis of Statistic Images from Functional Mapping Experiments , 1996, Journal of cerebral blood flow and metabolism : official journal of the International Society of Cerebral Blood Flow and Metabolism.

[8]  Carlo Bonferroni.  Sulle medie multiple di potenze , 1950 .

[9]  R. Blair,et al.  An exact statistical method for comparing topographic maps, with any number of subjects and electrodes , 2005, Brain Topography.

[10]  Mark Appelbaum,et al.  Bias in the Analysis of Repeated-Measures Designs: Some Alternative Approaches. , 1973 .

[11]  M. Cohen.  Analysis of Variance with Repeated Measures on Autonomic Responses , 1987 .

[12]  L. Trainor,et al.  Measuring temporal resolution in infants using mismatch negativity , 2001, Neuroreport.

[13]  S. Geisser,et al.  On methods in the analysis of profile data , 1959 .

[14]  Thomas E. Nichols,et al.  Nonparametric permutation tests for functional neuroimaging: A primer with examples , 2002, Human brain mapping.

[15]  R. McCarley,et al.  Analysis of scalp voltage asymmetries using Hotelling'sT2 methodology , 2005, Brain Topography.

[16]  E. Edgington.  Approximate Randomization Tests , 1969 .

[17]  J. Kenemans,et al.  Differential effects of amitriptyline, nefazodone and paroxetine on performance and brain indices of visual selective attention and working memory , 2002, Psychopharmacology.

[18]  H. Huynh,et al.  Conditions under Which Mean Square Ratios in Repeated Measurements Designs Have Exact F-Distributions , 1970 .

[19]  H. Keselman,et al.  Multiple Comparison Procedures , 2005 .

[20]  Gideon Keren,et al.  A Handbook for Data Analysis in the Behavioral Sciences: Statistical Issues , 1993 .

[21]  N C Andreasen,et al.  Tests for Comparing Images Based on Randomization and Permutation Methods , 1996, Journal of cerebral blood flow and metabolism : official journal of the International Society of Cerebral Blood Flow and Metabolism.

[22]  T. Micceri.  The unicorn, the normal curve, and other improbable creatures. , 1989 .

[23]  M. Yale,et al.  Getting the point: electrophysiological correlates of protodeclarative pointing , 2002, International Journal of Developmental Neuroscience.

[24]  R. Biscay,et al.  Testing topographic differences between event related brain potentials by using non-parametric combinations of permutation tests. , 1997, Electroencephalography and clinical neurophysiology.

[25]  G. Jogesh Babu,et al.  Multivariate Permutation Tests , 2002, Technometrics.

[26]  Olive Jean Dunn,et al.  Confidence Intervals for the Means of Dependent, Normally Distributed Variables , 1959 .

[27]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[28]  Karl-Heinz Jockel,et al.  Finite Sample Properties and Asymptotic Efficiency of Monte Carlo Tests , 1986 .

[29]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[30]  A. Hayes.  Permutation test is not distribution-free: Testing H₀: ρ = 0. , 1996 .

[31]  G F Potts,et al.  Dense sensor array topography of the event-related potential to task-relevant auditory stimuli. , 1998, Electroencephalography and clinical neurophysiology.

[32]  Rupert G. Miller.  Simultaneous Statistical Inference , 1966 .

[33]  J. Thayer,et al.  The continuing problem of false positives in repeated measures ANOVA in psychophysiology: a multivariate solution. , 1987, Psychophysiology.

[34]  Jacob Cohen.  The Cost of Dichotomization , 1983 .

[35]  J. Kromrey,et al.  Power and Type I Error Rates of New Pairwise Multiple Comparison Procedures Under Heterogeneous Variances , 1995 .

[36]  J. J. Higgins,et al.  A Study of Multivariate Permutation Tests Which May Replace Hotelling's T2 Test in Prescribed Circumstances. , 1994, Multivariate behavioral research.