The Use and Abuse of Transcranial Magnetic Stimulation to Modulate Corticospinal Excitability in Humans

The magnitude and direction of reported physiological effects induced using transcranial magnetic stimulation (TMS) to modulate human motor cortical excitability have proven difficult to replicate routinely. We conducted an online survey on the prevalence and possible causes of these reproducibility issues. A total of 153 researchers were identified via their publications and invited to complete an anonymous internet-based survey that asked about their experience trying to reproduce published findings for various TMS protocols. The prevalence of questionable research practices known to contribute to low reproducibility was also determined. We received 47 completed surveys from researchers with an average of 16.4 published papers (95% CI 10.8–22.0) that used TMS to modulate motor cortical excitability. Respondents also had a mean of 4.0 (2.5–5.7) relevant completed studies that would never be published. Across a range of TMS protocols, 45–60% of respondents found similar results to those in the original publications; the other respondents were able to reproduce the original effects only sometimes or not at all. Only 20% of respondents used formal power calculations to determine study sample sizes. Others relied on previously published studies (25%), personal experience (24%) or flexible post-hoc criteria (41%). Approximately 44% of respondents knew researchers who engaged in questionable research practices (range 32–70%), yet only 18% admitted to engaging in them (range 6–38%). These practices included screening subjects to find those that respond in a desired way to a TMS protocol, selectively reporting results and rejecting data based on a gut feeling. In a sample of 56 published papers that were inspected, not a single questionable research practice was reported. Our survey revealed that approximately 50% of researchers are unable to reproduce published TMS effects. Researchers need to start increasing study sample size and eliminating—or at least reporting—questionable research practices in order to make the outcomes of TMS research reproducible.

[1]  M. Ridding,et al.  Inter- and intra-subject variability of motor cortex plasticity following continuous theta-burst stimulation , 2015, Neuroscience.

[2]  Ulrich Dirnagl,et al.  Robust research: Institutions must do their part for reproducibility , 2015, Nature.

[3]  R. Kaplan,et al.  Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time , 2015, PloS one.

[4]  R. Lanfear,et al.  Evidence of Experimental Bias in the Life Sciences: Why We Need Blind Data Recording , 2015, PLoS biology.

[5]  Brian A. Nosek,et al.  Promoting an open research culture , 2015, Science.

[6]  Stephen E. Fienberg,et al.  Self-correction in science at work , 2015, Science.

[7]  B. Hemmelgarn,et al.  Exploring physician specialist response rates to web-based surveys , 2015, BMC Medical Research Methodology.

[8]  W. Anderson.  Stamp out shabby research conduct , 2015, Nature.

[9]  R. Lanfear,et al.  The Extent and Consequences of P-Hacking in Science , 2015, PLoS biology.

[10]  L. Halsey,et al.  The fickle P value generates irreproducible results , 2015, Nature Methods.

[11]  S. Rossi,et al.  Evidence-based guidelines on the therapeutic use of repetitive transcranial magnetic stimulation (rTMS) , 2014, Clinical Neurophysiology.

[12]  J. Rothwell,et al.  Corticospinal activity evoked and modulated by non‐invasive stimulation of the intact human motor cortex , 2014, The Journal of physiology.

[13]  Neil Malhotra,et al.  Publication bias in the social sciences: Unlocking the file drawer , 2014, Science.

[14]  M. Ridding,et al.  Non-invasive induction of plasticity in the human cortex: Uses and limitations , 2014, Cortex.

[15]  J. Mervis.  Research Transparency. Why null results rarely see the light of day. , 2014, Science.

[16]  B. Cheeran,et al.  Inter-individual Variability in Response to Non-invasive Brain Stimulation Paradigms , 2014, Brain Stimulation.

[17]  Regina Nuzzo,et al.  Scientific method: Statistical errors , 2014, Nature.

[18]  C. Gamble,et al.  Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias — An Updated Review , 2013, PloS one.

[19]  J. Rothwell,et al.  The role of interneuron networks in driving human motor cortical plasticity. , 2013, Cerebral cortex.

[20]  Leif D. Nelson,et al.  P-Curve: A Key to the File Drawer , 2013, Journal of experimental psychology. General.

[21]  Brian A. Nosek,et al.  Power failure: why small sample size undermines the reliability of neuroscience , 2013, Nature Reviews Neuroscience.

[22]  Rex B. Kline,et al.  Beyond Significance Testing: Statistics Reform in the Behavioral Sciences , 2013 .

[23]  J. Wicherts,et al.  The Rules of the Game Called Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[24]  H. Beek.  F1000Prime recommendation of False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. , 2012 .

[25]  C. Ferguson,et al.  Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. , 2012, Psychological methods.

[26]  G. Loewenstein,et al.  Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling , 2012, Psychological science.

[27]  E. Wagenmakers,et al.  Erroneous analyses of interactions in neuroscience: a problem of significance , 2011, Nature Neuroscience.

[28]  J. Schooler.  Unpublished results hide the decline effect , 2011, Nature.

[29]  Á. Pascual-Leone,et al.  Safety of Theta Burst Transcranial Magnetic Stimulation: A Systematic Review of the Literature , 2011, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[30]  M. Ridding,et al.  Determinants of the induction of cortical plasticity by non‐invasive brain stimulation in healthy subjects , 2010, The Journal of physiology.

[31]  H. Kölsch,et al.  Reporting bias in medical research - a narrative review , 2010, Trials.

[32]  J. Ioannidis.  Why Most Discovered True Associations Are Inflated , 2008, Epidemiology.

[33]  J. Ioannidis,et al.  Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias , 2008, PloS one.

[34]  J. Rothwell,et al.  Consensus: Motor cortex plasticity protocols , 2008, Brain Stimulation.

[35]  I. Cuthill,et al.  Effect size, confidence interval and statistical significance: a practical guide for biologists , 2007, Biological reviews of the Cambridge Philosophical Society.

[36]  Michael C. Ridding,et al.  Factors influencing the magnitude and reproducibility of corticomotor excitability changes induced by paired associative stimulation , 2007, Experimental Brain Research.

[37]  S. C. Gandevia,et al.  Theta burst stimulation does not reliably depress all regions of the human motor cortex , 2006, Clinical Neurophysiology.

[38]  J. Ioannidis,et al.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[39]  Melissa S. Anderson,et al.  Scientists behaving badly , 2005, Nature.

[40]  J. Rothwell,et al.  Theta Burst Stimulation of the Human Motor Cortex , 2005, Neuron.

[41]  Barbara M. Wildemuth,et al.  Pixels or pencils? The relative effectiveness of Web-based versus paper surveys , 2004 .

[42]  L. Cohen,et al.  Induction of plasticity in the human motor cortex by paired associative stimulation. , 2000, Brain : a journal of neurology.

[43]  B L Day,et al.  Delay in the execution of voluntary movement by electrical or magnetic brain stimulation in intact man. Evidence for the storage of motor programs in the brain. , 1989, Brain : a journal of neurology.

[44]  A. Barker,et al.  NON-INVASIVE MAGNETIC STIMULATION OF HUMAN MOTOR CORTEX , 1985, The Lancet.

[45]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[46]  T. Sterling.  Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa , 1959 .

[47]  Jan Lundqvist.  Why and How , 1947 .