The culture of orphaned texts: Academic books in a performance-based evaluation system

The purpose of this paper is to contribute to the body of knowledge on how research evaluation in different national and organisational contexts affects, often in unintended ways, research and publication practices. In particular, it looks at the development of book publication in the social sciences and humanities (SSH) in the Czech Republic since 2004, when a performance-based system of evaluation was introduced, up to the present.,The paper builds upon ethnographic research complemented by the analysis of Czech science policy documents, data available in the governmental database “Information Register of R&D results” and formal and informal interviews with expert evaluators and other stakeholders in the research system. It further draws on the authors’ own experience as scholars, who have also over the years participated in a number of evaluation procedures as peers and experts.,The number of books published by researchers in SSH based at Czech institutions has risen considerably in reaction to the pressure for productivity that is inscribed into the evaluation methodology and has resulted in the rise of in-house publishing by researchers’ own research institution, “fake internationalisation” using foreign low-quality presses as the publication venue, and the development of a culture of orphaned books that have no readers.,In the Czech Republic robust and internationally harmonised bibliometric data regarding books would definitely help to create a form of research evaluation that would stimulate meaningful scholarly book production. At the same time, better-resourced and better-designed peer review evaluation is needed.,This is the first attempt to analyse in detail the conditions and consequences the Czech performance-based research evaluation system has for SSH book publication. The paper demonstrates that often discussed harming of SSH and book-writing in particular by performance-based IF-centred research evaluation does not necessarily manifest in declining numbers of publications. On the contrary, the number of books published may increase at the cost of producing more texts of questionable scholarly quality.

[1]  A. Bonaccorsi Peer Review in Social Sciences and Humanities. Addressing the Interpretation of Quality Criteria , 2018 .

[2]  Björn Hammarfelt Beyond coverage : Toward a bibliometrics for the humanities , 2016 .

[3]  S. Rijcke,et al.  Bibliometrics: The Leiden Manifesto for research metrics , 2015, Nature.

[4]  Sherry B. Ortner Anthropology and Social Theory: Culture, Power, and the Acting Subject , 2006 .

[5]  Marilyn Strathern,et al.  Audit Cultures: Anthropological Studies in Accountability, Ethics and the Academy , 2000 .

[6]  Brigitte Tiefenthaler,et al.  Counting quality? The Czech performance-based research funding system , 2015 .

[7]  Paul Wouters,et al.  Evaluation practices and effects of indicator use : a literature review , 2016 .

[8]  M. Lamont,et al.  How Quality Is Recognized by Peer Review Panels: The Case of the Humanities , 2016 .

[9]  Tereza Stöckelová Immutable Mobiles Derailed: STS, Geopolitics, and Research Assessment , 2012 .

[10]  S. Rijcke,et al.  Bibliometrics: The Leiden Manifesto for research metrics. , 2015, Nature.

[11]  L. Butler,et al.  Explaining Australia’s increased share of ISI publications—the effects of a funding formula based on publication counts , 2003 .

[12]  Jochen Gläser,et al.  Governing Science , 2016, European Journal of Sociology.

[13]  Susan Leigh Star,et al.  Sorting Things Out: Classification and Its Consequences , 1999 .

[14]  W. V. D. Akker,et al.  Yes We Should : Research Assessment in the Humanities , 2016 .

[15]  C. Faggiolani,et al.  Mapping the Role of the Book in Evaluation at the Individual and Department Level in Italian SSH. A Multisource Analysis , 2018 .

[16]  Filip Vostal,et al.  Academic stratospheres-cum-underworlds: when highs and lows of publication cultures meet , 2017, Aslib J. Inf. Manag..

[17]  Hans-Dieter Daniel,et al.  Research assessment in the humanities: Introduction , 2016 .

[18]  Philippe Baveye,et al.  Battling the paper glut. , 2010, Science.

[19]  Tereza Stöckelová,et al.  Public accountability and the politicization of science: The peculiar journey of Czech research assessment , 2012 .

[20]  Bea Mahieu,et al.  International Audit of Research, Development & Innovation in the Czech Republic , 2011 .

[21]  Peter Ingwersen,et al.  Taking scholarly books into account: current developments in five European countries , 2016, Scientometrics.

[22]  Rens Bod,et al.  Can we rank scholarly book publishers? A bibliometric experiment with the field of history , 2015, J. Assoc. Inf. Sci. Technol..

[23]  Sarah de Rijcke,et al.  Implicated in the Indicator Game? An Experimental Debate , 2017 .

[24]  Andrea Bonaccorsi,et al.  The evaluation of research in Social Sciences and Humanities. Lessons from the Italian experience , 2018 .

[25]  Nicolás Robinson-García,et al.  Reviewing, indicating, and counting books for modern research evaluation systems , 2018, Springer Handbook of Science and Technology Indicators.

[26]  Tereza Stöckelová,et al.  Power at the Interfaces: The Contested Orderings of Academic Presents and Futures in a Social Science Department , 2014 .

[27]  Sarah de Rijcke,et al.  Thinking with Indicators. Exploring the Epistemic Impacts of Academic Performance Indicators in the Life Sciences , 2017 .