Inter-Rater Reliability of Preprocessing EEG Data: Impact of Subjective Artifact Removal on Associative Memory Task ERP Results

The processing of EEG data routinely involves subjective removal of artifacts during a preprocessing stage. Preprocessing inter-rater reliability (IRR) and how differences in preprocessing may affect outcomes of primary event-related potential (ERP) analyses has not been previously assessed. Three raters independently preprocessed EEG data of 16 cognitively healthy adult participants (ages 18–39 years) who performed a memory task. Using intraclass correlations (ICCs), IRR was assessed for Early-frontal, Late-frontal, and Parietal Old/new memory effects contrasts across eight regions of interest (ROIs). IRR was good to excellent for all ROIs; 22 of 26 ICCs were above 0.80. Raters were highly consistent in preprocessing across ROIs, although the frontal pole ROI (ICC range 0.60–0.90) showed less consistency. Old/new parietal effects had highest ICCs with the lowest variability. Rater preprocessing differences did not alter primary ERP results. IRR for EEG preprocessing was good to excellent, and subjective rater-removal of EEG artifacts did not alter primary memory-task ERP results. Findings provide preliminary support for robustness of cognitive/memory task-related ERP results against significant inter-rater preprocessing variability and suggest reliability of EEG to assess cognitive-neurophysiological processes multiple preprocessors are involved.

[1]  A. Daffertshofer,et al.  Effects of sleep deprivation on neural functioning: an integrative review , 2007, Cellular and Molecular Life Sciences.

[2]  S. Segalowitz,et al.  The effect of sleepiness on performance monitoring: I know what I am doing, but do I care? , 2006, Journal of sleep research.

[3]  T. Curran,et al.  Using ERPs to dissociate recollection from familiarity in picture recognition. , 2003, Brain research. Cognitive brain research.

[4]  J. Cavanagh,et al.  Mood effects on the ERP processing of emotional intensity in faces: a P3 investigation with depressed students. , 2006, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[5]  William O Tatum,et al.  Artifact and Recording Concepts in EEG , 2011, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[6]  Mark E. Pflieger,et al.  Theory of a Spatial Filter for Removing Ocular Artifacts With Preservation of EEG , 2001 .

[7]  Redmond G O'Connell,et al.  Retest reliability of event-related potentials: evidence from a variety of paradigms. , 2012, Psychophysiology.

[8]  Jeffrey D. Johnson,et al.  The relationship between the right frontal old/new ERP effect and post-retrieval monitoring: Specific or non-specific? , 2008, Neuropsychologia.

[9]  P Berg,et al.  A multiple source approach to the correction of eye artifacts. , 1994, Electroencephalography and clinical neurophysiology.

[10]  F. Perrin,et al.  Spherical splines for scalp potential and current density mapping. , 1989, Electroencephalography and clinical neurophysiology.

[11]  Marco Cecchi,et al.  A clinical trial to validate event-related potential markers of Alzheimer's disease in outpatient settings , 2015, Alzheimer's & dementia.

[12]  R. Knight,et al.  Error-Monitoring and Post-Error Compensations: Dissociation between Perceptual Failures and Motor Errors with and without Awareness , 2013, The Journal of Neuroscience.

[13]  Xu Lei,et al.  Understanding the Influences of EEG Reference: A Large-Scale Brain Network Perspective , 2017, Front. Neurosci..

[14]  Kaia L. Vilberg,et al.  Functional significance of retrieval‐related activity in lateral parietal cortex: Evidence from fMRI and ERPs , 2009, Human brain mapping.

[15]  T. Curran.  Brain potentials of recollection and familiarity , 2000, Memory & cognition.

[16]  M. Rugg,et al.  Probability effects on event-related potential correlates of recognition memory. , 2003, Brain research. Cognitive brain research.

[17]  Jürgen Kayser,et al.  In search of the Rosetta Stone for scalp EEG: Converging on reference-free techniques , 2010, Clinical Neurophysiology.

[18]  M. Rugg,et al.  Event-related potentials and recognition memory , 2007, Trends in Cognitive Sciences.

[19]  Tim Curran,et al.  The FN400 indexes familiarity-based recognition of faces , 2007, NeuroImage.

[20]  Filippo Zappasodi,et al.  Impact of the reference choice on scalp EEG connectivity estimation , 2016, Journal of neural engineering.

[21]  Kevin A Hallgren,et al.  Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. , 2012, Tutorials in quantitative methods for psychology.

[22]  D. Cicchetti.  Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology. , 1994 .

[23]  A Gevins,et al.  Test–retest reliability of cognitive EEG , 2000, Clinical Neurophysiology.

[24]  Ali K. Bourisly,et al.  Influence of sex on P300: an event-related potential electrophysiological study , 2016, Neuroreport.

[25]  D. Yao,et al.  A method to standardize a reference of scalp EEG recordings to a point at infinity , 2001, Physiological measurement.

[26]  Brandon A. Ally,et al.  The worth of pictures: Using high density event-related potentials to understand the memorial power of pictures and the dynamics of recognition memory , 2007, NeuroImage.

[27]  Paul L. Nunez,et al.  REST: A good idea but not the gold standard , 2010, Clinical Neurophysiology.

[28]  Meghan B. Mitchell,et al.  Recognition of faces and names: multimodal physiological correlates of memory and executive function , 2016, Brain Imaging and Behavior.

[29]  K. McGraw,et al.  Forming inferences about some intraclass correlation coefficients. , 1996 .