Is interactive open access publishing able to identify high-impact submissions? A study on the predictive validity of Atmospheric Chemistry and Physics by using percentile rank classes

In a comprehensive research project, we investigated the predictive validity of selection decisions and reviewers' ratings at the open access journal Atmospheric Chemistry and Physics (ACP). ACP is a high‐impact journal publishing papers on the Earth's atmosphere and the underlying chemical and physical processes. Scientific journals have to deal with the following question concerning the predictive validity: Are in fact the “best” scientific works selected from the manuscripts submitted? In this study we examined whether selecting the “best” manuscripts means selecting papers that after publication show top citation performance as compared to other papers in this research area. First, we appraised the citation impact of later published manuscripts based on the percentile citedness rank classes of the population distribution (scaling in a specific subfield). Second, we analyzed the association between the decisions (n = 677 accepted or rejected, but published elsewhere manuscripts) or ratings (reviewers' ratings for n = 315 manuscripts), respectively, and the citation impact classes of the manuscripts. The results confirm the predictive validity of the ACP peer review system. © 2011 Wiley Periodicals, Inc.

[1]  David M. Schultz,et al.  REjEctIOn RAtES FOR jOURnAlS PUBlISHInG In tHE AtMOSPHERIc ScIEncES , 2010 .

[2]  Peter Vinkler,et al.  Evaluation of some methods for the relative assessment of scientific publications , 1986, Scientometrics.

[3]  Claudio Castellano,et al.  Universality of citation distributions: Toward an objective measure of scientific impact , 2008, Proceedings of the National Academy of Sciences.

[4]  Lutz Bornmann,et al.  A multilevel modelling approach to investigating the predictive validity of editorial decisions: do the editors of a high profile journal select manuscripts that are highly cited after publication? , 2011 .

[5]  Sun Huh,et al.  Peer Review and Manuscript Management in Scientific Journals: Guidelines for Good Practice , 2008, Journal of Educational Evaluation for Health Professions.

[6]  A Vesterdal.  [The fate of manuscripts]. , 1991, Sygeplejersken.

[7]  Iain D. Craig,et al.  Do open access articles have greater citation impact?: A critical review of the literature , 2007, J. Informetrics.

[8]  Werner Marx,et al.  The publication and citation impact profiles of Angewandte Chemie and the Journal of the American Chemical Society based on the sections of Chemical Abstracts: A case study on the limitations of the Journal Impact Factor , 2009 .

[9]  Alan Agresti,et al.  Categorical Data Analysis , 2003 .

[10]  Lutz Bornmann,et al.  OPEN PEN ACCESS CCESS , 2008 .

[11]  S. Lock,et al.  A difficult balance: editorial peer review in medicine continued. , 1985 .

[12]  Irene Hames,et al.  Peer Review and Manuscript Management in Scientific Journals: Guidelines for Good Practice , 2008 .

[13]  Ulrich Pöschl,et al.  Interactive Open Access Publishing and Peer Review: The Effectiveness and Perspectives of Transparency and Self-Regulation in Scientific Communication and Evaluation , 2010 .

[14]  D. Aksnes.  CHARACTERISTICS OF HIGHLY CITED PAPERS , 2003 .

[15]  Ronald Rousseau,et al.  Median and percentile impact factors: A set of new indicators , 2005, Scientometrics.

[16]  Ronald N. Kostoff,et al.  Citation analysis of research performer quality , 2004, Scientometrics.

[17]  Hans-Dieter Daniel,et al.  A new reference standard for citation analysis in chemistry and related fields based on the sections of Chemical Abstracts , 2009, Scientometrics.

[18]  Ulrich Pöschl,et al.  Interactive journal concept for improved scientific publishing and quality assurance , 2004, Learn. Publ..

[19]  Henk F. Moed,et al.  Handbook of Quantitative Science and Technology Research , 2005 .

[20]  Hans-Dieter Daniel,et al.  Guardians of Science: Fairness and Reliability of Peer Review , 1994 .

[21]  T. Opthof,et al.  Regrets or no regrets? No regrets! The fate of rejected manuscripts. , 2000, Cardiovascular research.

[22]  Thed N. van Leeuwen,et al.  Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? , 2002, Scientometrics.

[23]  Lutz Bornmann,et al.  Universality of citation distributions–A validation of Radicchi et al.'s relative indicator c f = c-c 0 at the micro level using data from chemistry , 2009 .

[24]  Lutz Bornmann,et al.  The b index as a measure of scientific excellence. A promising supplement to the h index , 2007 .

[25]  Ronald N. Kostoff,et al.  The metrics of science and technology , 2001, Scientometrics.

[26]  姚萍,et al.  了解心理测验过程 = Understanding psychological testing , 2000 .

[27]  Va Arlington.  National Science Board. , 2010 .

[28]  D. Cicchetti.  The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation , 1991, Behavioral and Brain Sciences.

[29]  Lutz Bornmann,et al.  Reliability of reviewers' ratings when using public peer review: a case study , 2010, Learn. Publ..

[30]  D F Kallmes,et al.  Fate of Manuscripts Previously Rejected by the American Journal of Neuroradiology: A Follow-Up Analysis , 2009, American Journal of Neuroradiology.

[31]  Lutz Bornmann,et al.  Universality of citation distributions-A validation of Radicchi et al.'s relative indicator cf = c/c0 at the micro level using data from chemistry , 2009, J. Assoc. Inf. Sci. Technol..

[32]  Hans-Dieter Daniel,et al.  Guardians of science , 1993 .

[33]  Mario Cortina-Borja,et al.  Handbook of Parametric and Nonparametric Statistical Procedures, 5th edn , 2012 .

[34]  Jonathan Adams.  The use of bibliometrics to measure research quality in UK higher education institutions , 2009, Archivum Immunologiae et Therapiae Experimentalis.

[35]  Susan van Rooyen,et al.  The evaluation of peer‐review quality , 2001, Learn. Publ..

[36]  Edwin A. Henneken,et al.  Open Access does not increase citations for research articles from The Astrophysical Journal , 2007, ArXiv.

[37]  Werner Marx,et al.  The publication and citation impact profiles of Angewandte Chemie and the Journal of the American Chemical Society based on the sections of Chemical Abstracts: A case study on the limitations of the Journal Impact Factor , 2009, J. Assoc. Inf. Sci. Technol..

[38]  E. Giglia,et al.  Open access in the biomedical field: a unique opportunity for researchers (and research itself). , 2007, Europa medicophysica.

[39]  Theodora Bloom.  Online frontiers of the peer-reviewed literature , 2006 .

[40]  Lutz Bornmann,et al.  Selecting manuscripts for a high-impact journal through peer review: A citation analysis of communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere , 2008, J. Assoc. Inf. Sci. Technol..

[41]  Eduardo Figueredo.  The numerical equivalence between the impact factor of journals and the quality of the articles , 2006, J. Assoc. Inf. Sci. Technol..

[42]  J. Wilson.  Peer review and publication. Presidential address before the 70th annual meeting of the American Society for Clinical Investigation, San Francisco, California, 30 April 1978. , 1978, The Journal of clinical investigation.

[43]  R. Butler.  What good is research? , 1990, Geriatrics.

[44]  Finn Hansson,et al.  Research evaluation and organisational learning in the university: a possible coexistence? , 2006 .

[45]  A. Raan.  Measuring Science: Capita Selecta of Current Main Issues , 2004 .

[46]  Wolfgang Glänzel,et al.  An item-by-item subject classification of papers published in multidisciplinary and general journals using reference analysis , 2006, Scientometrics.

[47]  Peter Taylor,et al.  Citation Statistics , 2009, ArXiv.

[48]  D. Shatz.  Peer Review: A Critical Inquiry , 2004 .

[49]  P. Perakakis,et al.  The siege of science , 2008 .

[50]  Peter Gölitz.  Twitter, Facebook, and Open Access … , 2010 .

[51]  Peter Vinkler,et al.  Relations of relative scientometric impact indicators. The relative publication strategy index , 1997, Scientometrics.

[52]  Rex B. Kline,et al.  Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research , 2004 .

[53]  Nancy McCormack,et al.  Peer Review and Legal Publishing: What Law Librarians Need to Know about Open, Single-Blind, and Double-Blind Reviewing , 2009 .

[54]  Elizabeth A. Marschall,et al.  Exploring the Peer Review Process: What is it, Does it Work, and Can it Be Improved? , 2009 .

[55]  Irene Hames,et al.  Peer Review and Manuscript Management in Scientific Journals , 2007 .

[56]  Fytton Rowland,et al.  The peer‐review process , 2002, Learn. Publ..

[57]  Andreas Thor,et al.  From black box to white box at open access journals: predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics , 2010 .

[58]  David J. Sheskin,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 1997 .

[59]  Reinier Plomp,et al.  The significance of the number of highly cited papers as an indicator of scientific prolificacy , 1990, Scientometrics.

[60]  L. Bornmann,et al.  The effectiveness of the peer review process: inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. , 2008, Angewandte Chemie.

[61]  Tibor Braun,et al.  Cross-field normalization of scientometric indicators , 1996, Scientometrics.

[62]  Lutz Bornmann,et al.  Citation speed as a measure to predict the attention an article receives: An investigation of the validity of editorial decisions at Angewandte Chemie International Edition , 2010, J. Informetrics.

[63]  Graham Thornicroft,et al.  Fair assessment of the merits of psychiatric research , 2007, British Journal of Psychiatry.

[64]  Koenraad Debackere,et al.  Subfield-specific normalized relative indicators and a new generation of relational charts: Methodological foundations illustrated on the assessment of institutional research performance , 2008, Scientometrics.

[65]  Ulrich Pöschl,et al.  An open, two-stage peer-review journal , 2006 .

[66]  Amy M. Hightower,et al.  Science and Engineering Indicators , 1993 .

[67]  Henk F. Moed,et al.  Handbook of Quantitative Science and Technology Research: The Use of Publication and Patent Statistics in Studies of S&T Systems , 2004 .

[68]  Anthony F. J. van Raan,et al.  Advanced bibliometric methods for the evaluation of universities , 1999, Scientometrics.

[69]  R. Conroy,et al.  Choosing an Appropriate Real-Life Measure of Effect Size: The Case of a Continuous Predictor and a Binary Outcome , 2002 .

[70]  Thed N. van Leeuwen,et al.  Modelling of bibliometric approaches and importance of output verification in research performance assessment , 2007 .

[71]  W. G. Cochran.  Some Methods for Strengthening the Common χ 2 Tests , 1954 .