ERP Reliability Analysis (ERA) Toolbox: An open-source toolbox for analyzing the reliability of event-related brain potentials.

Generalizability theory (G theory) provides a flexible, multifaceted approach to estimating score reliability. G theory's approach to estimating score reliability has important advantages over classical test theory that are relevant for research using event-related brain potentials (ERPs). For example, G theory does not require parallel forms (i.e., equal means, variances, and covariances), can handle unbalanced designs, and provides a single reliability estimate for designs with multiple sources of error. This monograph provides a detailed description of the conceptual framework of G theory using examples relevant to ERP researchers, presents the algorithms needed to estimate ERP score reliability, and provides a detailed walkthrough of newly-developed software, the ERP Reliability Analysis (ERA) Toolbox, that calculates score reliability using G theory. The ERA Toolbox is open-source, Matlab software that uses G theory to estimate the contribution of the number of trials retained for averaging, group, and/or event types on ERP score reliability. The toolbox facilitates the rigorous evaluation of psychometric properties of ERP scores recommended elsewhere in this special issue.

[1]  D. Meyer,et al.  A Neural System for Error Detection and Compensation , 1993 .

[2]  Peter E. Clayson,et al.  How does noise affect amplitude and latency measurement of event-related potentials (ERPs)? A methodological critique and simulation study. , 2013, Psychophysiology.

[3]  P. Shrout.  Measurement reliability and agreement in psychiatry , 1998, Statistical methods in medical research.

[4]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[5]  J. Richard Jennings,et al.  Editorial Policy on Analyses of Variance With Repeated Measures , 1987 .

[6]  J. Hohnsbein,et al.  Effects of crossmodal divided attention on late ERP components. II. Error processing in choice reaction tasks. , 1991, Electroencephalography and clinical neurophysiology.

[7]  Jeffrey N. Rouder,et al.  The fallacy of placing confidence in confidence intervals , 2015, Psychonomic bulletin & review.

[8]  Klaas Sijtsma,et al.  Reliability Beyond Theory and Into Practice , 2008, Psychometrika.

[9]  M. Vince.  The intermittency of control movements and the psychological refractory period. , 1948, The British journal of psychology. General section.

[10]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[11]  S. Luck.  An Introduction to the Event-Related Potential Technique , 2005 .

[12]  Robert L. Brennan,et al.  An Essay on the History and Future of Reliability from the Perspective of Replications , 2001 .

[13]  Klaas Sijtsma,et al.  On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha , 2008, Psychometrika.

[14]  Peter E. Clayson,et al.  Making sense of all the conflict: a theoretical review and critique of conflict-related ERPs. , 2014, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[15]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[16]  S. Luck,et al.  The effects of electrode impedance on data quality and statistical significance in ERP recordings. , 2010, Psychophysiology.

[17]  John M Lachin,et al.  The role of measurement reliability in clinical trials , 2004, Clinical trials.

[18]  Peter E. Clayson,et al.  The dependability of electrophysiological measurements of performance monitoring in a clinical sample: A generalizability and decision analysis of the ERN and Pe. , 2015, Psychophysiology.

[19]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[20]  Scott A. Baldwin,et al.  Performance monitoring following conflict: Internal adjustments in cognitive control? , 2012, Neuropsychologia.

[21]  Edward H. Haertel,et al.  4 Reliability Coefficients and Generalizability Theory , 2006 .

[22]  J. Cacioppo,et al.  Handbook Of Psychophysiology , 2019 .

[23]  R. Brennan.  Generalizability Theory and Classical Test Theory , 2010 .

[24]  Peter E Clayson,et al.  Psychometric considerations in the measurement of event-related brain potentials: Guidelines for measurement and reporting. , 2017, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[25]  Robert M. O’Brien,et al.  Generalizability coefficients are reliability coefficients , 1995 .

[26]  J. Stroop.  Studies of interference in serial verbal reactions. , 1992 .

[27]  Richard J. Shavelson,et al.  Generalizability Theory: A Primer , 1991 .

[28]  M. W. Richardson,et al.  The theory of the estimation of test reliability , 1937 .

[29]  C. Eriksen,et al.  Effects of noise letters upon the identification of a target letter in a nonsearch task , 1974 .

[30]  Margot J. Taylor,et al.  Guidelines for using human event-related potentials to study cognition: recording standards and publication criteria. , 2000, Psychophysiology.

[31]  J. Weir.  Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. , 2005, Journal of strength and conditioning research.

[32]  Michael T. Kane,et al.  AN INDEX OF DEPENDABILITY FOR MASTERY TESTS , 1977 .