Evaluating the Impact of Science, Technology and Innovation Programs: a Methodological Toolkit

The purpose of this guideline is to provide ideas and technical advice on how to measure the effectiveness of Science, Technology and Innovation Programs (STIP). The paper addresses the specific challenges of evaluating STIP, from the assessment of the intervention logic to the choice of the most appropriate method to solve the attribution problem. Much attention is devoted to the topic of data, discussing pros and cons of different data sources, data quality issues, and strategies for data collection. The paper analyzes in detail the potential application of experimental and quasi-experimental methods to STIP. For each method, the paper highlights characteristics and assumptions, practical issues related to the implementation, and strengths and weakness specifically related to the application to STIP. Other specific issues related to the evaluation of STIP are also considered: the timing of effects, intensity of treatment, multiple treatments, impact heterogeneity, externalities, and general equilibrium effects. Concrete examples of rigorous evaluations of STIP support the discussion of the various topics throughout the guideline.

[1]  Mark D. R. Parsons,et al.  An Evaluation of the Federal Tax Credit for Scientific Research and Experimental Development , 2007 .

[2]  C. Glymour,et al.  STATISTICS AND CAUSAL INFERENCE , 1985 .

[3]  Gustavo Crespi,et al.  Measuring Innovation in Latin America: What we did, where we are and what we want to do , 2007 .

[4]  Bruno Van Pottelsberghe,et al.  Evaluation of current fiscal incentives for business R&D in Belgium , 2003 .

[5]  Daniel Chudnovsky,et al.  EVALUATING A PROGRAM OF PUBLIC FUNDING OF SCIENTIFIC ACTIVITY. A CASE STUDY OF FONCYT IN ARGENTINA , 2006 .

[6]  J. Stiglitz,et al.  Credit Rationing in Markets with Imperfect Information , 1981 .

[7]  Alessandro Maffioli,et al.  The Impact of Funding on Research Collaboration: Evidence from Argentina , 2010 .

[8]  R. Nelson.  Research on Productivity Growth and Productivity Differences: Dead Ends and New Departures , 1981 .

[9]  Markus Frölich,et al.  Unconditional Quantile Treatment Effects Under Endogeneity , 2013 .

[10]  I. Busom,et al.  Who participates in R&D subsidy programs? , 2004 .

[11]  Alessandro Maffioli,et al.  Evaluating the impact of technology development funds in emerging economies: evidence from Latin America , 2008 .

[12]  Bruno Van Pottelsberghe,et al.  The Economics of the European Patent System: IP Policy for Innovation and Competition , 2007 .

[13]  Eric Strobl,et al.  The Effect of R&D Subsidies on Private R&D , 2005 .

[14]  David Card,et al.  Minimum Wages and Employment: A Case Study of the Fast Food Industry in New Jersey and Pennsylvania , 1993 .

[15]  E. Andersen,et al.  National Systems of Innovation: Towards a Theory of Innovation and Interactive Learning , 1992 .

[16]  W. Edward Steinmueller,et al.  Economics of technology policy , 2010 .

[17]  Guido W. Imbens,et al.  Regression Discontinuity Designs: A Guide to Practice. NBER Working Paper No. 13039. , 2007 .

[18]  Oliviero Antonio Carboni,et al.  The Effect of R&D Subsidies on Private R&D: Evidence from Italian Manufacturing Data , 2008 .

[19]  G. Imbens,et al.  The Propensity Score with Continuous Treatments , 2005 .

[20]  J. Lerner.  The Government as Venture Capitalist: The Long-Run Impact of the Sbir Program , 1998 .

[21]  Raffaello Bronzini,et al.  Are Incentives for R&D Effective? Evidence from a Regression Discontinuity Approach , 2011 .

[22]  R. Nelson.  The Simple Economics of Basic Scientific Research , 1959, Journal of Political Economy.

[23]  Nicolas Serrano-Velarde,et al.  Crowding-Out At The Top: The Heterogeneous Impact of R&D Subsidies on Firm Investment , 2009 .

[24]  Bart Verspagen,et al.  Technology-gaps, innovation-diffusion and transformation:an evolutionary interpretation , 2002 .

[25]  Otto Toivanen,et al.  Innovation policy, entrepreneurship, and development: a Finnish view , 2009 .

[26]  Boris Lokshin,et al.  How effective are level-based R&D tax credits? Evidence from the Netherlands , 2012 .

[27]  Gustavo Crespi,et al.  Productivity Growth, Knowledge Flows, and Spillovers , 2007 .

[28]  Markus Frlich,et al.  Finite-Sample Properties of Propensity-Score Matching and Weighting Estimators , 2004, Review of Economics and Statistics.

[29]  Bruno Crépon,et al.  Research, Innovation, and Productivity: an Econometric Analysis at the Firm Level , 1998 .

[30]  K. Arrow.  Economic Welfare and the Allocation of Resources for Invention , 1962 .

[31]  Sebastian Galiani,et al.  Water for Life: The Impact of the Privatization of Water Services on Child Mortality , 2004, Journal of Political Economy.

[32]  Pierre Mohnen,et al.  Are Firms that Receive R&D Subsidies More Innovative? , 2009 .

[33]  Richard Blundell.  Dynamic panel data methods and practice , 2002 .

[34]  David S. Lee,et al.  Regression Discontinuity Designs in Economics , 2009 .

[35]  Joshua D. Angrist,et al.  Mostly Harmless Econometrics: An Empiricist's Companion , 2008 .

[36]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[37]  Saul Lach,et al.  Do R&D Subsidies Stimulate or Displace Private R&D? Evidence from Israel , 2000 .

[38]  Emmanuel Duguet,et al.  Are R&D Subsidies a Substitute or a Complement to Privately Funded R&D? Evidence from France Using Propensity Score Methods for Non-Experimental Data , 2003 .

[39]  S. Wallsten.  The Effects of Government-Industry R&D Programs on Private R&D: The Case of the Small Business Innovation Research Program , 2000 .

[40]  J. Heckman,et al.  Making the Most out of Programme Evaluations and Social Experiments: Accounting for Heterogeneity in Programme Impacts , 1997 .

[41]  Alessandro Maffioli,et al.  Public Support to Innovation: The Colombian COLCIENCIAS' Experience , 2011 .

[42]  Brian A. Jacob,et al.  The Impact of Research Grant Funding on Scientific Productivity , 2007, Journal of public economics.

[43]  F. Wolak,et al.  Structural Econometric Modeling: Rationales and Examples from Industrial Organization , 2004 .

[44]  Stephen Bond,et al.  Dynamic panel data models: a guide to micro data methods and practice , 2002 .

[45]  J. Angrist,et al.  Does Compulsory School Attendance Affect Schooling and Earnings? , 1990 .

[46]  David S. Lee,et al.  Program Evaluation and Research Designs. NBER Working Paper No. 16016. , 2010 .

[47]  Abhijit Banerjee,et al.  The Experimental Approach to Development Economics , 2008 .

[48]  Marco Caliendo,et al.  Some Practical Guidance for the Implementation of Propensity Score Matching , 2005, SSRN Electronic Journal.

[49]  Sudipto Bhattacharya,et al.  Abstract: Innovation and Communication: Signaling with Partial Disclosure , 1980 .

[50]  Andrew A. Toole,et al.  Is Public R&D a Complement or Substitute for Private R&D? A Review of the Econometric Evidence , 1999 .

[51]  Michael J. Cooper,et al.  Measuring innovation. , 2011, Healthcare executive.

[52]  G. Imbens,et al.  On the Failure of the Bootstrap for Matching Estimators , 2006 .

[53]  D. Czarnitzki,et al.  The Effects of Public R&D Subsidies on Firms' Innovation Activities , 2003 .

[54]  V. Chernozhukov,et al.  An IV Model of Quantile Treatment Effects , 2002 .

[55]  John T. Scott,et al.  The Nature of Innovation Market Failure and the Design of Public Support for Private Innovation , 2000 .

[56]  Karin Hoisl,et al.  Inventors and invention processes in Europe: Results from the PatVal-EU survey , 2007 .

[57]  Adam B. Jaffe,et al.  Building Programme Evaluation into the Design of Public Research‐Support Programmes , 2002 .

[58]  Josh Lerner,et al.  The Financing of R&D and Innovation , 2009 .

[59]  Almas Heshmati,et al.  The Impact of Public Funds on Private R&D Investment: New Evidence from a Firm Level Innovation Study ∗ , 2005 .

[60]  Carmen Pagés,et al.  The age of productivity : transforming economies from the bottom up , 2010 .

[61]  Richard W. Harris,et al.  Is a higher rate of R&D tax credit a panacea for low levels of R&D in disadvantaged regions? , 2009 .

[62]  Dirk Czarnitzki,et al.  Research and development: financial constraints and the role of public funding for small and medium-sized enterprises , 2002 .

[63]  M. Trevor.  Technology Policy and Economic Performance. Lessons from Japan , 1989 .

[64]  Manuel Trajtenberg,et al.  Innovation Policy for Development: an Overview , 2005 .

[65]  O. Ashenfelter,et al.  Estimating the Effect of Training Programs on Earnings , 1976 .

[66]  Josh Lerner,et al.  When Bureaucrats Meet Entrepreneurs: The Design of Effective 'Public Venture Capital' Programmes , 2002 .

[67]  I. Busom,et al.  Who Participates in R&D Subsidy Programs? The Case of Spanish Manufacturing Firms , 2004 .

[68]  Alessandro Maffioli,et al.  A Primer for Applying Propensity-Score Matching , 2010 .

[69]  Sergio Firpo.  Efficient Semiparametric Estimation of Quantile Treatment Effects , 2004 .

[70]  Petra E. Todd,et al.  Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme , 1997 .

[71]  G. Imbens,et al.  Better Late than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009) , 2009 .

[72]  Tommy Høyvarde Clausen,et al.  Do subsidies have positive impacts on R&D and innovation activities at the firm level? , 2009 .

[73]  Richard K. Crump,et al.  Dealing with limited overlap in estimation of average treatment effects , 2009 .

[74]  Daniel A. Levinthal,et al.  Innovation and Learning: The Two Faces of R&D , 1989 .

[75]  Alessandro Maffioli,et al.  Evaluating the Effectiveness of Public Support to Private R&D: Evidence from Argentina , 2006 .

[76]  R&D tax incentives: rationale, design, evaluation , 2011 .

[77]  J. Angrist,et al.  Instrumental Variables Estimates of the Effect of Subsidized Training on the Quantiles of Trainee Earnings , 1999 .

[78]  Mario Albornoz.  Conocimiento y desarrollo en América Latina: Antecedentes, evolución y perspectivas de las políticas de ciencia, tecnología e innovación , 2013 .

[79]  Alessandro Maffioli,et al.  Public Support to Firm-Level Innovation: An Evaluation of the FONTEC Program , 2012 .

[80]  Zvi Griliches,et al.  Issues in Assessing the Contribution of Research and Development to Productivity Growth , 1979 .

[81]  Jeffrey M. Woodbridge.  Econometric Analysis of Cross Section and Panel Data , 2002 .

[82]  María Consuelo Pazó Martínez,et al.  Do public subsidies stimulate private R&D spending? , 2006 .

[83]  M. Harding,et al.  A quantile regression approach for estimating panel data models using instrumental variables , 2009 .

[84]  B. Verspagen,et al.  Systems of Innovation , 2009 .