The statistical crisis in science: how is it relevant to clinical neuropsychology?

Abstract There is currently increased attention to the statistical (and replication) crisis in science. Biomedicine and social psychology have been at the heart of this crisis, but similar problems are evident in a wide range of fields. We discuss three examples of replication challenges from the field of social psychology and some proposed solutions, and then consider the applicability of these ideas to clinical neuropsychology. In addition to procedural developments such as preregistration and open data and criticism, we recommend that data be collected and analyzed with more recognition that each new study is a part of a learning process. The goal of improving neuropsychological assessment, care, and cure is too important to not take good scientific practice seriously.

[1]  Thomas Boraud,et al.  Replication Validity of Initial Association Studies: A Comparison between Psychiatry, Neurology and Four Somatic Diseases , 2016, PloS one.

[2]  Richard McElreath,et al.  The natural selection of bad science , 2016, Royal Society Open Science.

[3]  M. Baker.  1,500 scientists lift the lid on reproducibility , 2016, Nature.

[4]  N. Gates,et al.  A Neuropsychologist’s Guide To Undertaking a Systematic Review for Publication: Making the most of PRISMA Guidelines , 2016, Neuropsychology Review.

[5]  R. Grasman,et al.  Normative comparisons for large neuropsychological test batteries: User-friendly and sensitive solutions to minimize familywise false positives , 2016, Journal of clinical and experimental neuropsychology.

[6]  J. Zihl,et al.  The contribution of single case studies to the neuroscience of vision. , 2016, PsyCh journal.

[7]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[8]  J. Uri..  [37] Power Posing: Reassessing The Evidence Behind The Most Popular TED Talk , 2015 .

[9]  Roberto A. Weber,et al.  Assessing the Robustness of Power Posing , 2015, Psychological science.

[10]  Torkel Klingberg,et al.  Benefits of a Working Memory Training Program for Inattention in Daily Life: A Systematic Review and Meta-Analysis , 2015, PloS one.

[11]  A. Gelman.  The Connection Between Varying Treatment Effects and the Crisis of Unreplicable Research , 2015 .

[12]  Denny Borsboom,et al.  A Skeptical Eye on Psi , 2015 .

[13]  Mike R. Schoenberg,et al.  Introduction to the Special Issue on Improving Neuropsychological Research Through Use of Reporting Guidelines , 2014, The Clinical neuropsychologist.

[14]  Amy E. Sanders,et al.  Clinical Significance in Dementia Research , 2014, American journal of Alzheimer's disease and other dementias.

[15]  A. Gelman,et al.  The statistical crisis in science , 2014 .

[16]  R. Wiers,et al.  Prevalence and diagnostic validity of motivational impairments and deficits in visuospatial short-term memory and working memory in ADHD subtypes , 2014, European Child & Adolescent Psychiatry.

[17]  C. Jack,et al.  Biomarker Modeling of Alzheimer’s Disease , 2013, Neuron.

[18]  Michael J. Kofler,et al.  Do programs designed to train working memory, other executive functions, and attention benefit children with ADHD? A meta-analytic review of cognitive, academic, and behavioral outcomes. , 2013, Clinical psychology review.

[19]  J. Uderman,et al.  Cogmed Working Memory Training for Youth with ADHD: A Closer Examination of Efficacy Utilizing Evidence-Based Criteria , 2013, Journal of clinical child and adolescent psychology : the official journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53.

[20]  G. Francis.  Replication, statistical consistency, and publication bias. , 2013 .

[21]  Brian A. Nosek,et al.  Power failure: why small sample size undermines the reliability of neuroscience , 2013, Nature Reviews Neuroscience.

[22]  Steven W. Evans,et al.  Measuring ADHD and ODD Symptoms and Impairment Using High School Teachers' Ratings , 2013, Journal of clinical child and adolescent psychology : the official journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53.

[23]  Monica Melby-Lervåg,et al.  Is working memory training effective? A meta-analytic review. , 2013, Developmental psychology.

[24]  K. Goedert,et al.  Clock drawing in spatial neglect: a comprehensive analysis of clock perimeter, placement, and accuracy. , 2012, Journal of neuropsychology.

[25]  Randall W. Engle,et al.  Cogmed working memory training: Does the evidence support the claims? , 2012 .

[26]  A. Raz,et al.  Training the brain: Fact and fad in cognitive and behavioral remediation , 2012, Brain and Cognition.

[27]  Jeffrey R. Spies,et al.  Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability , 2012, 1205.4251.

[28]  E. Yong.  Replication studies: Bad copy , 2012, Nature.

[29]  K. Duff.  Evidence-based indicators of neuropsychological change in the individual patient: relevant concepts and methods. , 2012, Archives of clinical neuropsychology : the official journal of the National Academy of Neuropsychologists.

[30]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[31]  Axel Cleeremans,et al.  Behavioral Priming: It's All in the Mind, but Whose Mind? , 2012, PloS one.

[32]  D. Tranel,et al.  Neuropsychological assessment, 5th ed. , 2012 .

[33]  Jasiek Krzysztofiak.  IN THE WAKE OF HIGH-PROFILE CONTROVERSIES, PSYCHOLOGISTS ARE FACING UP TO PROBLEMS WITH REPLICATION. , 2012 .

[34]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[35]  Kenneth Hugdahl,et al.  Fifty years of dichotic listening research – Still going and going and… , 2011, Brain and Cognition.

[36]  J. Ioannidis,et al.  Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta-analyses. , 2011, JAMA.

[37]  Michael J. Taylor,et al.  Normative data and validation of a regression based summary score for assessing meaningful neuropsychological change , 2011, Journal of clinical and experimental neuropsychology.

[38]  Alexandra B. Morrison,et al.  Does working memory training work? The promise and challenges of enhancing cognition by training working memory , 2011, Psychonomic bulletin & review.

[39]  R. Schulz,et al.  Financial exploitation and psychological mistreatment among older adults: differences between African Americans and non-African Americans in a population-based survey. , 2010, The Gerontologist.

[40]  Susanne M. Jaeggi,et al.  Improving intelligence: a literature review. , 2010, Swiss medical weekly.

[41]  Amy J. C. Cuddy,et al.  Power Posing: Brief Nonverbal Displays Affect Neuroendocrine Levels and Risk Tolerance , 2010 .

[42]  H M Huizenga,et al.  Meta-analysis of CSF and MRI biomarkers for detecting preclinical Alzheimer's disease , 2009, Psychological Medicine.

[43]  H. Pashler,et al.  Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition 1 , 2009, Perspectives on psychological science : a journal of the Association for Psychological Science.

[44]  Andrew Gelman,et al.  Why We (Usually) Don't Have to Worry About Multiple Comparisons , 2009, 0907.2478.

[45]  Jacqueline Tetroe,et al.  Some theoretical underpinnings of knowledge translation. , 2007, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[46]  A. Gelman,et al.  The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant , 2006 .

[47]  Richard Smith,et al.  Peer Review: A Flawed Process at the Heart of Science and Journals , 2006, Journal of the Royal Society of Medicine.

[48]  P. Schatz,et al.  Misuse of statistical tests in Archives of Clinical Neuropsychology publications. , 2005, Archives of clinical neuropsychology : the official journal of the National Academy of Neuropsychologists.

[49]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[50]  H. Forssberg,et al.  Computerized training of working memory in children with ADHD--a randomized, controlled trial. , 2005, Journal of the American Academy of Child and Adolescent Psychiatry.

[51]  G. Cumming,et al.  Inference by eye: confidence intervals and how to read pictures of data. , 2005, The American psychologist.

[52]  S. Millis.  Statistical Practices: The Seven Deadly Sins , 2003, Child neuropsychology : a journal on normal and abnormal development in childhood and adolescence.

[53]  H. Forssberg,et al.  Training of Working Memory in Children With ADHD , 2002 .

[54]  R. Graves,et al.  Statistical Power and Effect Sizes of Clinical Neuropsychology Research , 2001, Journal of clinical and experimental neuropsychology.

[55]  W. Arrindell,et al.  Establishing clinically significant change: increment of precision and the distinction between individual and group level of analysis. , 1999, Behaviour research and therapy.

[56]  Amy Wenzel,et al.  One hundred years of forgetting: A quantitative description of retention , 1996 .

[57]  J. Bargh,et al.  Automaticity of social behavior: direct effects of trait construct and stereotype-activation on action. , 1996, Journal of personality and social psychology.

[58]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[59]  G. Chelune,et al.  'T scores for change': An illustration of a regression approach to depicting change in clinical neuropsychology , 1993 .

[60]  Colin M. Macleod.  Half a century of research on the Stroop effect: an integrative review. , 1991, Psychological bulletin.

[61]  O. Spreen,et al.  A Compendium of Neuropsychological Tests: Administration, Norms, and Commentary , 1991 .

[62]  S. Zeki,et al.  A century of cerebral achromatopsia. , 1990, Brain : a journal of neurology.

[63]  P. Meehl.  Why Summaries of Research on Psychological Theories are Often Uninterpretable , 1990 .

[64]  P. Meehl.  Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. , 1978 .

[65]  A. Tversky,et al.  BELIEF IN THE LAW OF SMALL NUMBERS , 1971, Pediatrics.

[66]  E. Tulving,et al.  Memory and Verbal Learning , 1970 .