Questionable research practices among italian research psychologists

A survey in the United States revealed that an alarmingly large percentage of university psychologists admitted having used questionable research practices that can contaminate the research literature with false positive and biased findings. We conducted a replication of this study among Italian research psychologists to investigate whether these findings generalize to other countries. All the original materials were translated into Italian, and members of the Italian Association of Psychology were invited to participate via an online survey. The percentages of Italian psychologists who admitted to having used ten questionable research practices were similar to the results obtained in the United States although there were small but significant differences in self-admission rates for some QRPs. Nearly all researchers (88%) admitted using at least one of the practices, and researchers generally considered a practice possibly defensible if they admitted using it, but Italian researchers were much less likely than US researchers to consider a practice defensible. Participants’ estimates of the percentage of researchers who have used these practices were greater than the self-admission rates, and participants estimated that researchers would be unlikely to admit it. In written responses, participants argued that some of these practices are not questionable and they have used some practices because reviewers and journals demand it. The similarity of results obtained in the United States, this study, and a related study conducted in Germany suggest that adoption of these practices is an international phenomenon and is likely due to systemic features of the international research and publication processes.

[1]  D. Fanelli How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data , 2009, PloS one.

[2]  J. Wicherts,et al.  The (mis)reporting of statistical results in psychology journals , 2011, Behavior research methods.

[3]  J. Ioannidis Why Most Published Research Findings Are False , 2005, PLoS medicine.

[4]  R. Kievit,et al.  The meaning of “ signi fi cance ” for different types of research [ translated and annotated , 2014 .

[5]  Rex B. Kline,et al.  Beyond Significance Testing: Statistics Reform in the Behavioral Sciences , 2013 .

[6]  L. Guttman A basis for scaling qualitative data. , 1944 .

[7]  Han L. J. van der Maas,et al.  Science Perspectives on Psychological an Agenda for Purely Confirmatory Research on Behalf Of: Association for Psychological Science , 2022 .

[8]  K. Fiedler,et al.  Questionable Research Practices Revisited , 2016 .

[9]  U. Schimmack The ironic effect of significant results on the credibility of multiple-study articles. , 2012, Psychological methods.

[10]  Brian A. Nosek,et al.  Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention , 2014, Trends in Cognitive Sciences.

[11]  Michèle B. Nuijten,et al.  Standard analyses fail to show that US studies overestimate effect sizes in softer research , 2014, Proceedings of the National Academy of Sciences of the United States of America.

[12]  H. Pashler,et al.  Editors’ Introduction to the Special Section on Replicability in Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[13]  Neil Malhotra,et al.  Underreporting in Psychology Experiments , 2016 .

[14]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[15]  Melissa S. Anderson,et al.  Scientists behaving badly , 2005, Nature.

[16]  A. D. de Groot,et al.  The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas] , 2014 .

[17]  Jacob Cohen,et al.  THINGS I HAVE LEARNED (SO FAR) , 1990 .

[18]  J. Wicherts,et al.  Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results , 2011, PloS one.

[19]  G. Francis,et al.  Excess Success for Psychology Articles in the Journal Science , 2014, PloS one.

[20]  Denny Borsboom,et al.  PsychDisclosure.org , 2013, Perspectives on psychological science : a journal of the Association for Psychological Science.

[21]  G. Banks,et al.  The Chrysalis Effect , 2017 .

[22]  Brian A. Nosek,et al.  Recommendations for Increasing Replicability in Psychology † , 2013 .

[23]  Michèle B. Nuijten,et al.  The prevalence of statistical reporting errors in psychology (1985–2013) , 2015, Behavior Research Methods.

[24]  Jelte M. Wicherts,et al.  Conducting Meta-Analyses Based on p Values , 2016, Perspectives on psychological science : a journal of the Association for Psychological Science.

[25]  R. Kievit,et al.  The meaning of "significance" for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. 1969. , 2014, Acta psychologica.

[26]  E. Wagenmakers A practical solution to the pervasive problems ofp values , 2007, Psychonomic bulletin & review.

[27]  Mercer Jennifer Ann,et al.  PUBLICATION manual of the American Psychological Association. , 1952, Psychological bulletin.

[28]  Daniel Sarewitz,et al.  The pressure to publish pushes down quality , 2016, Nature.

[29]  Greg Guest,et al.  Using Guttman Scaling to Rank Wealth: Integrating Quantitative and Qualitative Data , 2000 .

[30]  Melissa S. Anderson,et al.  Normative Dissonance in Science: Results from a National Survey of U.S. Scientists , 2007, Journal of empirical research on human research ethics : JERHRE.

[31]  J. Ioannidis,et al.  US studies may overestimate effect sizes in softer research , 2013, Proceedings of the National Academy of Sciences.

[32]  Daniele Fanelli,et al.  Negative results are disappearing from most disciplines and countries , 2011, Scientometrics.

[33]  Sarah Necker Scientific misbehavior in economics , 2014 .

[34]  Hendrik P. van Dalen,et al.  Intended and Unintended Consequences of a Publish-or-Perish Culture: A Worldwide Survey , 2012, J. Assoc. Inf. Sci. Technol..

[35]  Joseph R. Rausch,et al.  Sample size planning for statistical power and accuracy in parameter estimation. , 2008, Annual review of psychology.

[36]  Michèle B. Nuijten,et al.  Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science , 2014, PloS one.

[37]  Write less, write well , 2015, Cortex.

[38]  S. Maxwell The persistence of underpowered studies in psychological research: causes, consequences, and remedies. , 2004, Psychological methods.

[39]  Moreno Marzolla Quantitative analysis of the Italian National Scientific Qualification , 2015, J. Informetrics.

[40]  D. Fanelli “Positive” Results Increase Down the Hierarchy of the Sciences , 2010, PloS one.

[41]  J. Wicherts,et al.  The Rules of the Game Called Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[42]  L. HARKing: Hypothesizing After the Results are Known , 2002 .

[43]  Jelte M. Wicherts,et al.  Researchers’ Intuitions About Power in Psychological Research , 2016, Psychological science.

[44]  Anselm L. Strauss,et al.  Basics of qualitative research : techniques and procedures for developing grounded theory , 1998 .

[45]  N. Hoffart Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory , 2000 .

[46]  Eric Medvet,et al.  Bibliometric Evaluation of Researchers in the Internet Age , 2014, Inf. Soc..

[47]  C. Chambers Registered Reports: A new publishing initiative at Cortex , 2013, Cortex.

[48]  Elizabeth Gilbert,et al.  Reproducibility Project: Results (Part of symposium called "The Reproducibility Project: Estimating the Reproducibility of Psychological Science") , 2014 .

[49]  R. Landis,et al.  Editorial: Evidence on Questionable Research Practices: The Good, the Bad, and the Ugly , 2016 .

[50]  Kate E Decleene,et al.  Publication Manual of the American Psychological Association , 2011 .

[51]  Introduction to the Special Section on Methods: Odds and End , 2015, Perspectives on psychological science : a journal of the Association for Psychological Science.

[52]  Brian A. Nosek,et al.  Scientific Utopia , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[53]  T. Arrison Responsible Science: Ensuring the Integrity of the Research Process , 2014 .

[54]  G. Loewenstein,et al.  Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling , 2012, Psychological science.

[55]  E. Wagenmakers,et al.  Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011). , 2011, Journal of personality and social psychology.