Comparing Different Methods for Multiple Testing in Reaction Time Data

Scientific research often deals with the problem of performing many tests of significance. However, this practice results in an increase of the likelihood of committing one or more Type I errors, which grows as the number of tests increases (e.g., Keselman, Cribbie, & Holland, 1999). In the most common approach, error rate is familywise controlled (Familywise Error Rate, FWER) by reducing the α value as a direct function of the number of comparisons to be computed. In the classic Bonferroni method (1936), the threshold probability ( FW α , usually

[1]  A Heathcote,et al.  Response time distributions and the Stroop Task: a test of the Cohen, Dunbar, and McClelland (1990) model. , 1992, Journal of experimental psychology. Human perception and performance.

[2]  P. Jolicoeur,et al.  A Solution to the Effect of Sample Size on Outlier Elimination , 1994 .

[3]  Craig Leth-Steensen,et al.  Decomposing the problem-size effect: A comparison of response time distributions across cultures , 2002, Memory & cognition.

[4]  R. Ratcliff.  Group reaction time distributions and an analysis of distribution statistics. , 1979, Psychological bulletin.

[5]  Andrew Heathcote,et al.  RTSYS: A DOS application for the analysis of reaction time data , 1996 .

[6]  Julia Kastner,et al.  Introduction to Robust Estimation and Hypothesis Testing , 2005 .

[7]  Frank Bretz,et al.  Recent developments in multiple comparison procedures , 2004 .

[8]  T. Zandt.  Analysis of Response Time Distributions , 2002 .

[9]  Philip H. Ramsey.  Power Differences between Pairwise Multiple Comparisons , 1978 .

[10]  Charles W. Dunnett,et al.  Power and sample size comparisons of stepwise FWE and FDR controlling test procedures in the normal many-one case , 2004 .

[11]  David J. Madden,et al.  Aging and Recognition Memory: Changes in Regional Cerebral Blood Flow Associated with Components of Reaction Time Distributions , 1999, Journal of Cognitive Neuroscience.

[12]  D. Mewhort,et al.  Analysis of Response Time Distributions: An Example Using the Stroop Task , 1991 .

[13]  D. Munoz,et al.  Inhibitory control of eye movements during oculomotor countermanding in adults with attention-deficit hyperactivity disorder , 2003, Experimental Brain Research.

[14]  Robert A. Cribbie,et al.  The pairwise multiple comparison multiplicity problem: An alternative approach to familywise and comparison wise Type I error control. , 1999 .

[15]  W. Schwarz.  The ex-Wald distribution as a descriptive model of response times , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[16]  T. Zandt,et al.  How to fit a response time distribution , 2000, Psychonomic bulletin & review.

[17]  D J Mewhort,et al.  The extralist-feature effect: evidence against item matching in short-term recognition memory. , 2000, Journal of experimental psychology. General.

[18]  Yoav Benjamini,et al.  Identifying differentially expressed genes using false discovery rate controlling procedures , 2003, Bioinform..

[19]  Yongchao Ge.  Resampling-based Multiple Testing for Microarray Data Analysis , 2003 .

[20]  R. Proctor,et al.  The influence of irrelevant location information on performance: A review of the Simon and spatial Stroop effects , 1995, Psychonomic bulletin & review.

[21]  D. Balota,et al.  Word frequency, repetition, and lexicality effects in word recognition tasks: beyond measures of central tendency. , 1999, Journal of experimental psychology. General.

[22]  R. Ratcliff,et al.  Retrieval Processes in Recognition Memory , 1976 .

[23]  R. Hohle.  INFERRED COMPONENTS OF REACTION TIMES AS FUNCTIONS OF FOREPERIOD DURATION. , 1965, Journal of experimental psychology.

[24]  R. Wilcox.  Applying Contemporary Statistical Techniques , 2003 .

[25]  C Leth-Steensen,et al.  Mean response times, variability, and skew in the responding of ADHD children: a response time distributional approach. , 2000, Acta psychologica.

[26]  Takuji Nishimura,et al.  Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator , 1998, TOMC.

[27]  R. Luce,et al.  Evidence from auditory simple reaction times for both change and level detectors , 1982, Perception & psychophysics.

[28]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[29]  S. Dudoit,et al.  Multiple Hypothesis Testing in Microarray Experiments , 2003 .

[30]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[31]  F. Craik,et al.  Lapses of Intention and Performance Variability Reveal Age-Related Increases in Fluctuations of Executive Control , 2002, Brain and Cognition.

[32]  J. Booth,et al.  Resampling-Based Multiple Testing. , 1994 .

[33]  Roger Ratcliff,et al.  A Theory of Memory Retrieval. , 1978 .

[34]  R. Ratcliff.  Methods for dealing with reaction time outliers. , 1993, Psychological bulletin.

[35]  Marcie Penner-Wilger,et al.  Calculation latency: The μ of memory and the τ of transformation , 2006 .

[36]  Jonathan Grainger,et al.  Is global shape sufficient for automatic object identification? , 2001 .

[37]  David Thissen,et al.  Quick and Easy Implementation of the Benjamini-Hochberg Procedure for Controlling the False Positive Rate in Multiple Comparisons , 2002 .

[38]  S. Andrews,et al.  Distinguishing common and task-specific processes in word identification: a matter of some moment? , 2001, Journal of experimental psychology. Learning, memory, and cognition.

[39]  Y. Benjamini,et al.  Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics , 1999 .

[40]  D. Balota,et al.  Levels of selective attention revealed through analyses of response time distributions. , 2000, Journal of experimental psychology. Human perception and performance.

[41]  Raymond Klein,et al.  Inhibition of return , 2000, Trends in Cognitive Sciences.

[42]  J. H. Neely.  Semantic priming effects in visual word recognition: A selective review of current findings and theories. , 1991 .