Academic criteria for appointment, promotion and rewards in medical research: where's the evidence?

All academic institutions use some criteria for faculty appointment, promotion and other rewards (e.g. major awards and recognitions). However, what is the evidence for these criteria? How consistently are they applied? How fair are they? What are their adverse effects? Eventually do they foster better science and do they lead biomedical research towards improving life expectancy and quality? Answers to these questions are not obvious. Some criteria are explicit in faculty policies. Some institutions use specific quantitative metrics with or without minimum ‘thresholds’ (see Table 1). Most use qualitative phrasing open to subjective interpretations. One suspects that some widely used criteria are not openly acknowledged in otherwise very lengthy, often heterogeneous, official documents. Criteria may also be directly or indirectly imposed by outside organizations (e.g. national research agencies or funders who control infrastructure and funds). It has been argued that incentive systems should be multidimensional, including productivity, quality, reproducibility, sharing and translation potential (‘PQRST’) [1], but many current systems weight productivity particularly heavily. These systems directly affect the volume and indirectly the quality of the scientific publication record. This was recognized at least as far back as the 1980s, with a proposal that promotion committees consider only a handful of a scientist’s publications, in the hopes of improving the quality of our ‘large and largely trivial’ literature [2]. After three decades of explosive growth, the informativeness of both the clinical and preclinical research literature is a subject of even greater concern [3]. The impact that productivity has on the appointment and promotion process is in part responsible for ‘salami-slicing’, authorship inflation and the dilution of the meaning of authorship itself [4]. Scientists should of course aspire to be productive. However, simple measures of productivity, such as raw numbers of publications, can be misleading or easily manipulated (‘gamed’) [1,5]. The number of papers published in journals with a high impact factor is more difficult to manipulate, but only a small proportion of the papers published in these journals have a major impact and cumulatively many more papers published elsewhere also have a major impact [6]. The number of extremely highly cited papers is very difficult to game and offers more assurance about the influence of the work in the relevant scientific field. But even then, one should still place this highly influential work in context. Was it replicated? Did it result in major progress? Was it refuted? [7] For example, several extremely highly cited papers on stem cells have been misleading and the retracted Wakefield paper linking MMR and autism has been cited 2216 times (Google Scholar). For very young scientists, publication and citation metrics are based on a very limited time frame and may not be very informative for initial appointments [8]. Institutions seek letters from peer experts for qualitative insights into candidates’ corpus of work. However, the extent to which peer selection is unbiased and representative, and assessments are truly informative – that is, based on a deep enough understanding of the candidate’s work and of the field – is unknown. To ensure that publications are fit for purpose [9] might require more attention of institutional promotion criteria to quantifiable attributes that can ensure that published science is credible, real progress is being made, and eventually health is improved. Many specific improvements could be targeted. For example, the inability to replicate methods and results is a serious problem in clinical and preclinical research. Results from reanalyses can be very different from the original findings [10]. Such replication checks can only be carried out when researchers share their methods and data. Publishing articles in which the methods are described in sufficient detail to enable their replication and access to the data and programming code for verifying results is currently not rewarded in academic medicine. Not surprisingly, data and code sharing is the exception in most biomedical science, with the exception of few domains like genomics. Several initiatives have been started to help improve the publication record[11]. The Alltrials campaign is trying to make more clinical trial results accessible. Trial registries, such as clinicaltrials.gov, can serve a similar purpose and also help reduce the prevalence of reporting biases, such as selective outcome reporting. The Open Science Collaboration is trying to counter the inability to replicate Methods and Results by