Measuring the productivity of national R&D systems : challenges in cross-national comparisons of R&D input and publication output indicators

We investigate methodological problems in measuring research productivity at the national level by comparing official R&D statistics from the OECD with publication data from the Web of Science. Data from 18 countries are analysed. This paper problematizes the approach taken in studies where R&D statistics are used as an input variable and publications as an output variable to draw conclusions about the productivity or efficiency of national research and innovation systems. We consider possible pitfalls in such analyses and propose steps that can improve the comparability of these two data sources. We recommend that efforts are made among the OECD member countries to produce more reliable and commensurable international R&D statistics.

[1]  Organización para la Cooperación y el Desarrollo Económicos. Ocde.  Frascati manual 2015 : guidelines for collecting and reporting data in research and experimental development: the measurement of scientific, technological and innovation activities. , 2016 .

[2]  Gunnar Sivertsen,et al.  Patterns of internationalization and criteria for research assessment in the social sciences and humanities , 2016, Scientometrics.

[3]  Dag W. Aksnes,et al.  Publication rate expressed by age, gender and academic position - A large-scale analysis of Norwegian academic staff , 2015, J. Informetrics.

[4]  Concepts and definitions for identifying R&D (Frascati Manual 2015: Guidelines for Collecting and Reporting Data on Research and Experimental Development) , 2015 .

[5]  T. V. Leeuwen.  Bibliometric research evaluations, Web of Science and the Social Sciences and Humanities: a problematic relationship? , 2013 .

[6]  Fredrik Niclas Piro,et al.  A macro analysis of productivity differences across fields: Challenges in the measurement of scientific publishing , 2013, J. Assoc. Inf. Sci. Technol..

[7]  Juha Hedman,et al.  Productivity analysis of research in Natural Sciences, Technology and Clinical Medicine: an input–output model applied in comparison of Top 300 ranked universities of 4 North European and 4 East Asian countries , 2013, Scientometrics.

[8]  Dag W. Aksnes,et al.  Ranking national research systems by citation indicators. A comparative analysis using whole and fractionalised counting methods , 2012, J. Informetrics.

[9]  Birger Larsen,et al.  Comprehensive bibliographic coverage of the social sciences and humanities in a citation index: an empirical analysis of the potential , 2011, Scientometrics.

[10]  Thed N. van Leeuwen,et al.  Severe language effect in university rankings: particularly Germany and France are wronged in citation-based rankings , 2010, Scientometrics.

[11]  Loet Leydesdorff,et al.  Macro-level indicators of the relations between research funding and research output , 2009, J. Informetrics.

[12]  Peder Olesen Larsen,et al.  Comparisons of results of publication counting using different methods , 2008, Scientometrics.

[13]  Giovanni Abramo,et al.  The measurement of Italian universities’ research productivity by a non parametric-bibliometric methodology , 2008, Scientometrics.

[14]  Benoît Godin,et al.  Science, accounting and statistics: The input–output framework , 2007 .

[15]  E. Berndt,et al.  Introduction to "Hard-to-Measure Goods and Services: Essays in Honor of Zvi Griliches" , 2007 .

[16]  Jack E. Triplett,et al.  Title : Hard-to-Measure Goods and Services : Essays in Honor of Zvi Griliches , 2007 .

[17]  Van Ark,et al.  Hard-to-measure goods and services: Essays in memory of Zvi Griliches , 2007 .

[18]  B. Ark,et al.  International Comparisons of R&D Expenditure , 2007 .

[19]  Vincent Larivière,et al.  Benchmarking scientific output in the social sciences and humanities: The limits of existing databases , 2006, Scientometrics.

[20]  P. Zhou,et al.  The Emergence of China as a Leading Nation in Science. Research Policy, 35(1), 2006, 83-104. , 2006, 0911.3421.

[21]  Ronald Rousseau,et al.  The scientific wealth of European nations: Taking effectiveness into account , 1998, Scientometrics.

[22]  Barry Bozeman,et al.  The Impact of Research Collaboration on Scientific Productivity , 2005 .

[23]  Henk F. Moed,et al.  Handbook of Quantitative Science and Technology Research , 2005 .

[24]  D. King.  The scientific impact of nations , 2004, Nature.

[25]  Thed N. van Leeuwen,et al.  The Holy Grail of science policy: Exploring and combining bibliometric tools in search of scientific excellence , 2003, Scientometrics.

[26]  Thed N. van Leeuwen,et al.  Language biases in the coverage of the Science Citation Index and its consequencesfor international comparisons of national research performance , 2001, Scientometrics.

[27]  Marc Luwel.  The Use of Input Data in the Performance Analysis of R&D Systems , 2004 .

[28]  R. Kostoff.  The (scientific) wealth of nations , 2004 .

[29]  D. A. King,et al.  The Scientific Impact of Nations: What different countries get for their research spending , 2004 .

[30]  A. Raan.  Measuring Science: Capita Selecta of Current Main Issues , 2004 .

[31]  R. May.  The Scientific Investments of Nations , 1998, Science.

[32]  C. Martin,et al.  SA science and technology indicators , 1994 .

[33]  Amy M. Hightower,et al.  Science and Engineering Indicators , 1993 .

[34]  D. Antiseri.  [Scientific imagination]. , 1977, Medicina nei secoli.

[35]  Henk F. Moed,et al.  Citation Analysis in Research Evaluation , 1899 .