Test–retest reliability of evoked BOLD signals from a cognitive–emotive fMRI test battery

Even more than in cognitive research applications, moving fMRI to the clinic and the drug development process requires the generation of stable and reliable signal changes. The performance characteristics of the fMRI paradigm constrain experimental power and may require different study designs (e.g., crossover vs. parallel groups), yet fMRI reliability characteristics can be strongly dependent on the nature of the fMRI task. The present study investigated both within-subject and group-level reliability of a combined three-task fMRI battery targeting three systems of wide applicability in clinical and cognitive neuroscience: an emotional (face matching), a motivational (monetary reward anticipation) and a cognitive (n-back working memory) task. A group of 25 young, healthy volunteers were scanned twice on a 3T MRI scanner with a mean test-retest interval of 14.6 days. FMRI reliability was quantified using the intraclass correlation coefficient (ICC) applied at three different levels ranging from a global to a localized and fine spatial scale: (1) reliability of group-level activation maps over the whole brain and within targeted regions of interest (ROIs); (2) within-subject reliability of ROI-mean amplitudes and (3) within-subject reliability of individual voxels in the target ROIs. Results showed robust evoked activation of all three tasks in their respective target regions (emotional task=amygdala; motivational task=ventral striatum; cognitive task=right dorsolateral prefrontal cortex and parietal cortices) with high effect sizes (ES) of ROI-mean summary values (ES=1.11-1.44 for the faces task, 0.96-1.43 for the reward task, 0.83-2.58 for the n-back task). Reliability of group level activation was excellent for all three tasks with ICCs of 0.89-0.98 at the whole brain level and 0.66-0.97 within target ROIs. Within-subject reliability of ROI-mean amplitudes across sessions was fair to good for the reward task (ICCs=0.56-0.62) and, dependent on the particular ROI, also fair-to-good for the n-back task (ICCs=0.44-0.57) but lower for the faces task (ICC=-0.02-0.16). In conclusion, all three tasks are well suited to between-subject designs, including imaging genetics. When specific recommendations are followed, the n-back and reward task are also suited for within-subject designs, including pharmaco-fMRI. The present study provides task-specific fMRI reliability performance measures that will inform the optimal use, powering and design of fMRI studies using comparable tasks.

[1]  Charles R. G. Guttmann,et al.  Functional MRI of auditory verbal working memory: long-term reproducibility analysis , 2004, NeuroImage.

[2]  S. Rauch,et al.  Test-retest reliability of a functional MRI working memory paradigm in normal and schizophrenic subjects. , 2001, The American journal of psychiatry.

[3]  Matthias J. Wieser,et al.  Auditory cortex activation is modulated by emotion: A functional near-infrared spectroscopy (fNIRS) study , 2011, NeuroImage.

[4]  M. Reuter,et al.  Imaging gene–substance interactions: The effect of the DRD2 TaqIA polymorphism and the dopamine agonist bromocriptine on the brain activation during the anticipation of reward , 2006, Neuroscience Letters.

[5]  Andreas Meyer-Lindenberg,et al.  Genetic evidence implicating DARPP-32 in human frontostriatal structure, function, and cognition. , 2007, The Journal of clinical investigation.

[6]  J M Bland,et al.  Statistical methods for assessing agreement between two methods of clinical measurement , 1986 .

[7]  S. Strother,et al.  Reproducibility of BOLD‐based functional MRI obtained at 4 T , 1999, Human brain mapping.

[8]  I. Tracey,et al.  The role of fMRI in drug discovery , 2006, Journal of magnetic resonance imaging : JMRI.

[9]  W. Schneider,et al.  Neuroimaging studies of practice-related change: fMRI and meta-analytic evidence of a domain-general control network for learning. , 2005, Brain research. Cognitive brain research.

[10]  Catherine J. Harmer,et al.  Effect of a single dose of citalopram on amygdala response to emotional faces , 2009, British Journal of Psychiatry.

[11]  Axel Schäfer,et al.  Hemodynamic Effects of Negative Emotional Pictures – A Test-Retest Analysis , 2004, Neuropsychobiology.

[12]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[13]  R. Coppola,et al.  Functional Magnetic Resonance Imaging Brain Mapping in Psychiatry: Methodological Issues Illustrated in a Study of Working Memory in Schizophrenia , 1998, Neuropsychopharmacology.

[14]  N Jon Shah,et al.  Assessment of reliability in functional imaging studies , 2003, Journal of magnetic resonance imaging : JMRI.

[15]  F. Woermann,et al.  Amygdala fMRI lateralizes temporal lobe epilepsy , 2006, Neurology.

[16]  Lee Friedman,et al.  Report on a multicenter fMRI quality assurance protocol , 2006, Journal of magnetic resonance imaging : JMRI.

[17]  R. Cox,et al.  Test-retest precision of functional MR in sensory and motor task activation. , 1996, AJNR. American journal of neuroradiology.

[18]  Francesco Fera,et al.  The Amygdala Response to Emotional Stimuli: A Comparison of Faces and Scenes , 2002, NeuroImage.

[19]  K. Amunts,et al.  Cytoarchitectonic mapping of the human amygdala, hippocampal region and entorhinal cortex: intersubject variability and probability maps , 2005, Anatomy and Embryology.

[20]  P. Cheng,et al.  Bridging Functional MR Images and Scientific Inference: Reproducibility Maps , 2003, Journal of Cognitive Neuroscience.

[21]  Andreas Meyer-Lindenberg,et al.  Epistasis between catechol-O-methyltransferase and type II metabotropic glutamate receptor 3 genes on working memory brain function , 2007, Proceedings of the National Academy of Sciences.

[22]  D. Altman,et al.  Measuring agreement in method comparison studies , 1999, Statistical methods in medical research.

[23]  A. Beckett,et al.  AKUFO AND IBARAPA. , 1965, Lancet.

[24]  Guy M. Goodwin,et al.  Antidepressant Drug Treatment Modifies the Neural Processing of Nonconscious Threat Cues , 2006, Biological Psychiatry.

[25]  Andreas Meyer-Lindenberg,et al.  From maps to mechanisms through neuroimaging of schizophrenia , 2010, Nature.

[26]  F Barkhof,et al.  fMRI of visual encoding: Reproducibility of activation , 1999, Human brain mapping.

[27]  Felix Breuer,et al.  Neural response to reward anticipation is modulated by Gray's impulsivity , 2009, NeuroImage.

[28]  R. Müller,et al.  A critical discussion of intraclass correlation coefficients. , 1994, Statistics in medicine.

[29]  Giuseppe Blasi,et al.  Neural mechanisms of genetic risk for impulsivity and violence in humans. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[30]  A. Meyer-Lindenberg,et al.  Intermediate phenotypes and genetic mechanisms of psychiatric disorders , 2006, Nature Reviews Neuroscience.

[31]  Ian Marshall,et al.  Functional Magnetic Resonance Imaging (fMRI) reproducibility and variance components across visits and scanning sites with a finger tapping task , 2010, NeuroImage.

[32]  A. Hariri,et al.  Genetic variation in components of dopamine neurotransmission impacts ventral striatal reactivity associated with impulsivity , 2009, Molecular Psychiatry.

[33]  Daniel H. Mathalon,et al.  Using Brain Imaging Measures in Studies of Procognitive Pharmacologic Agents in Schizophrenia: Psychometric and Quality Assurance Considerations , 2011, Biological Psychiatry.

[34]  S. Cichon,et al.  Neural Mechanisms of a Genome-Wide Supported Psychosis Variant , 2009, Science.

[35]  N F Ramsey,et al.  Neurophysiological factors in human information processing capacity. , 2003, Brain : a journal of neurology.

[36]  Ann-Christine Ehlis,et al.  Enhancement of activity of the primary visual cortex during processing of emotional stimuli as measured with event‐related functional near‐infrared spectroscopy and event‐related potentials , 2008, Human brain mapping.

[37]  David Borsook,et al.  A procedural framework for good imaging practice in pharmacological fMRI studies applied to drug development #2: protocol optimization and best practices. , 2011, Drug discovery today.

[38]  Rebecca Elliott,et al.  The Effect of Citalopram Pretreatment on Neuronal Responses to Neuropsychological Tasks in Normal Volunteers: An fMRI Study , 2005, Neuropsychopharmacology.

[39]  Peter Kirsch,et al.  Anticipation of reward in a nonaversive differential conditioning paradigm and the brain reward system: an event-related fMRI study , 2003, NeuroImage.

[40]  René Hurlemann,et al.  Modulating amygdala responses to emotion: Evidence from pharmacological fMRI , 2011, Neuropsychologia.

[41]  A. Meyer-Lindenberg,et al.  5-HTTLPR polymorphism impacts human cingulate-amygdala interactions: a genetic susceptibility mechanism for depression , 2005, Nature Neuroscience.

[42]  David Borsook,et al.  A procedural framework for good imaging practice in pharmacological fMRI studies applied to drug development #1: processes and requirements. , 2011, Drug discovery today.

[43]  David Borsook,et al.  A role for fMRI in optimizing CNS drug development , 2006, Nature Reviews Drug Discovery.

[44]  J. Fleiss.  The design and analysis of clinical experiments , 1987 .

[45]  Steven C. R. Williams,et al.  Measuring fMRI reliability with the intra-class correlation coefficient , 2009, NeuroImage.

[46]  Tom Johnstone,et al.  Stability of amygdala BOLD response to fearful faces over multiple scan sessions , 2005, NeuroImage.

[47]  C. Terwee,et al.  When to use agreement versus reliability measures. , 2006, Journal of clinical epidemiology.

[48]  R. Dahl,et al.  Reward-related brain function as a predictor of treatment response in adolescents with major depressive disorder , 2010, Cognitive, affective & behavioral neuroscience.

[49]  Theo Gasser,et al.  Assessing intrarater, interrater and test–retest reliability of continuous measurements , 2002, Statistics in medicine.

[50]  A. Hariri,et al.  Temporal stability of individual differences in amygdala reactivity. , 2007, The American journal of psychiatry.

[51]  Michael B. Miller,et al.  How reliable are the results from functional magnetic resonance imaging? , 2010, Annals of the New York Academy of Sciences.

[52]  D. Glahn,et al.  Beyond hypofrontality: A quantitative meta‐analysis of functional neuroimaging studies of working memory in schizophrenia , 2005, Human brain mapping.

[53]  K. Lesch,et al.  Neural Hyporesponsiveness and Hyperresponsiveness During Immediate and Delayed Reward Processing in Adult Attention-Deficit/Hyperactivity Disorder , 2009, Biological Psychiatry.

[54]  Klaus Fliessbach,et al.  Retest reliability of reward-related BOLD signals , 2010, NeuroImage.

[55]  D. Altman,et al.  A note on the use of the intraclass correlation coefficient in the evaluation of agreement between two methods of measurement. , 1990, Computers in biology and medicine.

[56]  Sebastian Heinzel,et al.  Association between reward‐related activation in the ventral striatum and trait reward sensitivity is moderated by dopamine transporter genotype , 2011, Human brain mapping.

[57]  Richard G Wise,et al.  What is the value of human FMRI in CNS drug development? , 2010, Drug discovery today.

[58]  Brian Knutson,et al.  Ventral Striatal Hyporesponsiveness During Reward Anticipation in Attention-Deficit/Hyperactivity Disorder , 2007, Biological Psychiatry.

[59]  S. Rombouts,et al.  Within-subject reproducibility of visual activation patterns with functional magnetic resonance imaging using multislice echo planar imaging. , 1998, Magnetic resonance imaging.

[60]  T. V. van Erp,et al.  Reproducibility of visual activation during checkerboard stimulation in functional magnetic resonance imaging at 4 Tesla. , 2001, Japanese journal of ophthalmology.

[61]  Nick F. Ramsey,et al.  Test–retest reliability of fMRI activation during prosaccades and antisaccades , 2007, NeuroImage.

[62]  S. Rombouts,et al.  Test-retest analysis with functional MR of the activated area in the human visual cortex. , 1997, AJNR. American journal of neuroradiology.

[63]  Andreas Schulze-Bonhage,et al.  The reliability of fMRI activations in the medial temporal lobes in a verbal episodic memory task , 2005, NeuroImage.

[64]  P. Thompson,et al.  Heritability of Working Memory Brain Activation , 2011, The Journal of Neuroscience.

[65]  M. Bradley,et al.  Emotional arousal and activation of the visual cortex: an fMRI analysis. , 1998, Psychophysiology.

[66]  Edward W. Hsu,et al.  Reliability of fMRI motor tasks in structures of the corticostriatal circuitry: Implications for future studies and circuit function , 2010, NeuroImage.

[67]  Nathalie Tzourio-Mazoyer,et al.  Impact of cognitive performance on the reproducibility of fMRI activation in schizophrenia. , 2010, Journal of psychiatry & neuroscience : JPN.

[68]  Katharina Tabbert,et al.  Attention and amygdala activity: an fMRI study with spider pictures in spider phobia , 2009, Journal of Neural Transmission.