Reliability of EEG Measures of Interaction: A Paradigm Shift Is Needed to Fight the Reproducibility Crisis

Measures of interaction (connectivity) of the EEG are at the forefront of current neuroscientific research. Unfortunately, test-retest reliability can be very low, depending on the measure and its estimation, the EEG-frequency of interest, the length of the signal, and the population under investigation. In addition, artifacts can hamper the continuity of the EEG signal, and in some clinical situations it is impractical to exclude artifacts. We aimed to examine factors that moderate test-retest reliability of measures of interaction. The study involved 40 patients with a range of neurological diseases and memory impairments (age median: 60; range 21–76; 40% female; 22 mild cognitive impairment, 5 subjective cognitive complaints, 13 temporal lobe epilepsy), and 20 healthy controls (age median: 61.5; range 23–74; 70% female). We calculated 14 measures of interaction based on the multivariate autoregressive model from two EEG-recordings separated by 2 weeks. We characterized test-retest reliability by correlating the measures between the two EEG-recordings for variations of data length, data discontinuity, artifact exclusion, model order, and frequency over all combinations of channels and all frequencies, individually for each subject, yielding a correlation coefficient for each participant. Excluding artifacts had strong effects on reliability of some measures, such as classical, real valued coherence (~0.1 before, ~0.9 after artifact exclusion). Full frequency directed transfer function was highly reliable and robust against artifacts. Variation of data length decreased reliability in relation to poor adjustment of model order and signal length. Variation of discontinuity had no effect, but reliabilities were different between model orders, frequency ranges, and patient groups depending on the measure. Pathology did not interact with variation of signal length or discontinuity. Our results emphasize the importance of documenting reliability, which may vary considerably between measures of interaction. We recommend careful selection of measures of interaction in accordance with the properties of the data. When only short data segments are available and when the signal length varies strongly across subjects after exclusion of artifacts, reliability becomes an issue. Finally, measures which show high reliability irrespective of the presence of artifacts could be extremely useful in clinical situations when exclusion of artifacts is impractical.

[1]  Edward T. Bullmore,et al.  Reproducibility of graph metrics of human brain functional networks , 2009, NeuroImage.

[2]  Dorothee Auer,et al.  Reproducibility of Graph-Theoretic Brain Network Metrics: A Systematic Review , 2014, Brain Connect..

[3]  T. Gasser,et al.  Test-retest reliability of spectral parameters of the EEG. , 1985, Electroencephalography and clinical neurophysiology.

[4]  Luiz A. Baccalá,et al.  Partial directed coherence: a new concept in neural structure determination , 2001, Biological Cybernetics.

[5]  Katarzyna J. Blinowska,et al.  Determination of EEG activity propagation: pair-wise versus multichannel estimate , 2004, IEEE Transactions on Biomedical Engineering.

[6]  Clemens Brunner,et al.  BioSig: A Free and Open Source Software Library for BCI Research , 2008, Computer.

[7]  June Sic Kim,et al.  How reliable are the functional connectivity networks of MEG in resting states? , 2011, Journal of neurophysiology.

[8]  Katarzyna J. Blinowska,et al.  A new method of the description of the information flow in the brain structures , 1991, Biological Cybernetics.

[9]  Andreas Keil,et al.  Reliability of event-related EEG functional connectivity during visual entrainment: magnitude squared coherence and phase synchrony estimates. , 2015, Psychophysiology.

[10]  Habib Bousleiman,et al.  Reproducibility of Functional Connectivity and Graph Measures Based on the Phase Lag Index (PLI) and Weighted Phase Lag Index (wPLI) Derived from High Resolution EEG , 2014, PloS one.

[11]  R. Emmerson,et al.  Life-span changes in EEG spectral amplitude, amplitude variability and mean frequency , 1999, Clinical Neurophysiology.

[12]  M. Kramer,et al.  Emergence of Stable Functional Networks in Long-Term Human Electroencephalography , 2012, The Journal of Neuroscience.

[13]  R. Petersen,et al.  Mild cognitive impairment , 2006, The Lancet.

[14]  D Galin,et al.  EEG power spectra in normal and dyslexic children. I. Reliability during passive conditions. , 1983, Electroencephalography and clinical neurophysiology.

[15]  M. Kaminski,et al.  Determination of information flow direction among brain structures by a modified directed transfer function (dDTF) method , 2003, Journal of Neuroscience Methods.

[16]  Ulman Lindenberger,et al.  Peak individual alpha frequency qualifies as a stable neurophysiological trait marker in healthy younger and older adults. , 2013, Psychophysiology.

[17]  F. Hatz,et al.  Reliability of fully automated versus visually controlled pre- and post-processing of resting-state EEG , 2015, Clinical Neurophysiology.

[18]  Alois Schlögl,et al.  Analyzing event-related EEG data with multivariate autoregressive parameters. , 2006, Progress in brain research.

[19]  Georgios D. Mitsis,et al.  On the Effect of Volume Conduction on Graph Theoretic Measures of Brain Networks in Epilepsy , 2013 .

[20]  B. Oken,et al.  Test-retest reliability in EEG frequency analysis. , 1991, Electroencephalography and clinical neurophysiology.

[21]  Mingzhou Ding,et al.  Evaluating causal relations in neural systems: Granger causality, directed transfer function and statistical assessment of significance , 2001, Biological Cybernetics.

[22]  Antonio Napolitano,et al.  Test-retest reliability of graph metrics of resting state MRI functional brain networks: A review , 2015, Journal of Neuroscience Methods.

[23]  Andreas Heinz,et al.  Test–retest reliability of resting-state connectivity network characteristics using fMRI and graph theoretical measures , 2012, NeuroImage.

[24]  Moors Pieter,et al.  Test-retest reliability. , 2014 .

[25]  Mingzhou Ding,et al.  Cortical functional network organization from autoregressive modeling of local field potential oscillations , 2007, Statistics in medicine.

[26]  Marc Wildi,et al.  Test–retest reliability of resting EEG spectra validates a statistical signature of persons , 2007, Clinical Neurophysiology.

[27]  F. Babiloni,et al.  Estimation of the cortical functional connectivity with the multimodal integration of high-resolution EEG and fMRI data by directed transfer function , 2005, NeuroImage.

[28]  B. W. van Dijk,et al.  Opportunities and methodological challenges in EEG and MEG resting state functional brain network research , 2015, Clinical Neurophysiology.

[29]  Alois Schlögl,et al.  A comparison of multivariate autoregressive estimators , 2006, Signal Process..

[30]  W. M. Carey,et al.  Digital spectral analysis: with applications , 1986 .

[31]  B. Reisberg,et al.  The Global Deterioration Scale for assessment of primary degenerative dementia. , 1982, The American journal of psychiatry.

[32]  V. K. Murthy.  Estimation of the Cross-Spectrum , 1963 .

[33]  Markus R. Owen,et al.  Assessing cortico-hippocampal functional connectivity under anesthesia and kainic acid using generalized partial directed coherence , 2010, Biological Cybernetics.

[34]  Steven J. Schiff,et al.  Dangerous phase , 2007, Neuroinformatics.

[35]  A. Uhl,et al.  Reliability of EEG Interactions Differs between Measures and Is Specific for Neurological Diseases , 2017, Front. Hum. Neurosci..

[36]  Debora R. Baldwin,et al.  Reliability of quantitative EEG (qEEG) measures and LORETA current source density at 30 days , 2012, Neuroscience Letters.

[37]  Peter Fuhr,et al.  Microstate connectivity alterations in patients with early Alzheimer’s disease , 2015, Alzheimer's Research & Therapy.

[38]  Michael Eichler,et al.  On the Evaluation of Information Flow in Multivariate Systems by the Directed Transfer Function , 2006, Biological Cybernetics.

[39]  Wolfgang Klimesch,et al.  Resting state alpha frequency is associated with menstrual cycle phase, estradiol and use of oral contraceptives , 2014, Brain Research.

[40]  Karen O. Egiazarian,et al.  Measuring directional coupling between EEG sources , 2008, NeuroImage.

[41]  Kathleen S. Rockland,et al.  About connections , 2015, Front. Neuroanat..

[42]  John M Lachin,et al.  The role of measurement reliability in clinical trials , 2004, Clinical trials.

[43]  J. Fermaglich.  Electric Fields of the Brain: The Neurophysics of EEG , 1982 .

[44]  J. Geweke,et al.  Measurement of Linear Dependence and Feedback between Multiple Time Series , 1982 .

[45]  Barry Horwitz,et al.  The elusive concept of brain connectivity , 2003, NeuroImage.

[46]  L.A. Baccald,et al.  Generalized Partial Directed Coherence , 2007, 2007 15th International Conference on Digital Signal Processing.

[47]  Karl J. Friston,et al.  Evaluation of different measures of functional connectivity using a neural mass model , 2004, NeuroImage.

[48]  Hans Hallez,et al.  Accurate epileptogenic focus localization through time-variant functional connectivity analysis of intracranial electroencephalographic signals , 2011, NeuroImage.

[49]  Alvaro Pascual-Leone,et al.  Reliability of Resting-State Microstate Features in Electroencephalography , 2014, PloS one.

[50]  W. Gersch,et al.  Epileptic Focus Location: Spectral Analysis Method , 1970, Science.

[51]  Edward T. Bullmore,et al.  Validation of non-REM sleep stage decoding from resting state fMRI using linear support vector machines , 2016, NeuroImage.

[52]  A Gevins,et al.  Test–retest reliability of cognitive EEG , 2000, Clinical Neurophysiology.

[53]  M. Hallett,et al.  Identifying true brain interaction from EEG data using the imaginary part of coherency , 2004, Clinical Neurophysiology.

[54]  M. Baker.  1,500 scientists lift the lid on reproducibility , 2016, Nature.

[55]  R. E. Greenblatt,et al.  Connectivity measures applied to human brain electrophysiological data , 2012, Journal of Neuroscience Methods.

[56]  J. Geweke,et al.  Measures of Conditional Linear Dependence and Feedback between Time Series , 1984 .