Letting the daylight in: Reviewing the reviewers and other ways to maximize transparency in science

With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses.

[1]  Linda S. Gottfredson,et al.  Lessons in academic freedom as lived experience , 2010 .

[2]  Bruce Alberts,et al.  Making Data Maximally Available , 2011, Science.

[3]  Anthony F. J. van Raan,et al.  Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises , 1996, Scientometrics.

[4]  James R Murphy,et al.  Statistical errors in immunologic research. , 2004, The Journal of allergy and clinical immunology.

[5]  Karl P. Pfeiffer,et al.  The Use of Statistics in Medical Research , 2007 .

[6]  M. J. Bayarri,et al.  Calibration of ρ Values for Testing Precise Null Hypotheses , 2001 .

[7]  David Berle,et al.  Inconsistencies between reported test statistics and p‐values in two psychiatry journals , 2007, International journal of methods in psychiatric research.

[8]  Satoshi Kanazawa,et al.  General intelligence as a domain-specific adaptation. , 2004, Psychological review.

[9]  A. Vickers,et al.  Empirical Study of Data Sharing by Authors Publishing in PLoS Journals , 2009, PloS one.

[10]  Sung-Tae Hong,et al.  EDITOR'S NOTE - About This Supplement , 2007, Journal of Korean Medical Science.

[11]  E. García‐Berthou,et al.  Incongruence between test statistics and P values in medical papers , 2004 .

[12]  Alan S. Gerber,et al.  Publication Bias in Empirical Sociological Research , 2008 .

[13]  J. Wicherts,et al.  The (mis)reporting of statistical results in psychology journals , 2011, Behavior research methods.

[14]  J. Philippe Rushton,et al.  Brain size as an explanation of national differences in IQ, longevity, and other life-history variables , 2010 .

[15]  Kate E Decleene,et al.  Publication Manual of the American Psychological Association , 2011 .

[16]  Earl Hunt,et al.  Considerations Relating to the Study of Group Differences in Intelligence , 2007, Perspectives on psychological science : a journal of the Association for Psychological Science.

[17]  Richard Smith,et al.  Peer Review: A Flawed Process at the Heart of Science and Journals , 2006, Journal of the Royal Society of Medicine.

[18]  Alan S. Gerber,et al.  Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals , 2008 .

[19]  Jack Silknitter.  The Ups and Downs , 1935 .

[20]  Richard W. Smith.  In Search Of an Optimal Peer Review System , 2009 .

[21]  D. Benos,et al.  The ups and downs of peer review. , 2007, Advances in physiology education.

[22]  Joseph S. Rossi,et al.  How Often are Our Statistics Wrong? a Statistics Class Exercise , 1987 .

[23]  W. K. Simmons,et al.  Circular analysis in systems neuroscience: the dangers of double dipping , 2009, Nature Neuroscience.

[24]  Julien Mayor,et al.  Are Scientists Nearsighted Gamblers? The Misleading Nature of Impact Factors , 2010, Front. Psychology.

[25]  Donald I. Templer.  Can’t see the forest because of the trees , 2010 .

[26]  H. Pashler,et al.  Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition 1 , 2009, Perspectives on psychological science : a journal of the Association for Psychological Science.

[27]  A. Heinrichs.  Ups and downs. , 2001, Trends in molecular medicine.

[28]  J. Wicherts,et al.  Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results , 2011, PloS one.

[29]  Denny Borsboom,et al.  Why national IQs do not support evolutionary theories of intelligence , 2010 .

[30]  Thomas N. Sherratt,et al.  Statistical inference and spatial patterns in correlates of IQ , 2011 .

[31]  A. Wakefield,et al.  Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. , 1998, Lancet.

[32]  Richard Lynn,et al.  IQ and Global Inequality , 2006 .

[33]  M. Nowak,et al.  The evolution of eusociality , 2010, Nature.

[34]  Jelte M. Wicherts,et al.  Publish (your data) or (let the data) perish! Why not publish your data too? , 2012 .

[35]  Laurent Lehmann,et al.  Inclusive fitness theory and eusociality , 2011, Nature.

[36]  S. Oppenheimer.  Out of Eden , 2003 .

[37]  F. Godlee,et al.  Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. , 1998, JAMA.

[38]  J. Ioannidis,et al.  An exploratory test for an excess of significant findings , 2007, Clinical trials.

[39]  J. Ridley,et al.  An unexpected influence of widely used significance thresholds on the distribution of reported P‐values , 2007, Journal of evolutionary biology.

[40]  Denny Borsboom,et al.  Evolution, brain size, and the national IQ of peoples around 3000 years B.C , 2010 .

[41]  Satoshi Kanazawa,et al.  Temperature and evolutionary novelty as forces behind the evolution of general intelligence , 2008 .

[42]  Terri Gullickson,et al.  Publication Manual of the American Psychological Association (4th ed.). , 1996 .

[43]  Karl P. Pfeiffer,et al.  [MEDICINE] The Use of Statistics in Medical Research: A Comparison of The New England Journal of Medicine and Nature Medicine , 2007 .

[44]  Jerilee Grandy.  The New Know-Nothings: The Political Foes of the Scientific Study of Human Nature , 2000 .

[45]  E. Wagenmakers,et al.  Erroneous analyses of interactions in neuroscience: a problem of significance , 2011, Nature Neuroscience.

[46]  Gina Geffen,et al.  Temperature , Skin Color , Per Capita Income , and IQ : An International Perspective , 2003 .

[47]  Jelte M. Wicherts,et al.  Psychology must learn a lesson from fraud case , 2011, Nature.

[48]  Richard Lynn,et al.  Consistency of race differences in intelligence over millennia: A comment on Wicherts, Borsboom and Dolan , 2010 .

[49]  D. Borsboom,et al.  The poor availability of psychological research data for reanalysis. , 2006, The American psychologist.

[50]  B. Latané,et al.  Bystander intervention in emergencies: diffusion of responsibility. , 1968, Journal of personality and social psychology.

[51]  A. Anbar,et al.  A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus , 2011, Science.

[52]  Garry A. Gelade,et al.  The Geography of IQ. , 2008 .

[53]  F. Godlee,et al.  Wakefield’s article linking MMR vaccine and autism was fraudulent , 2011, BMJ : British Medical Journal.