Bringing rigour to energy innovation policy evaluation

Clean energy innovation is pivotal for low-cost energy sector decarbonization. Substantial public research and development funding is spent on energy innovation. Generating more evidence on which support mechanisms most effectively drive clean energy innovations, and why, could improve their design moving forward. In this Perspective, we discuss five challenges that researchers often face when attempting to rigorously evaluate energy innovation policies and public subsidy programmes. We recommend solutions, such as developing new innovation outcome metrics that consider unique features of the energy sector and building databases that cover long time periods. We also suggest that researchers and funding agencies work together to implement randomized control trials or conduct quasi-experimental evaluation of existing programmes and policies wherever possible. Evaluating the impact of renewable energy innovation research funding is important to design more productive research and development programmes. Pless et al. discuss the key challenges in effective evaluation of research funding outcomes and identify solutions.

[1]  Philippe Gagnepain,et al.  European Cooperative R&D and Firm Performance: Evidence Based on Funding Differences in Key Actions , 2013 .

[2]  R. Kemp,et al.  Technology and Environmental Policy , 2013 .

[3]  E. Rohlfing ARPA-E Impacts: A Sampling of Project Outcomes, Volume II , 2017 .

[4]  P. Aghion,et al.  Carbon Taxes, Path Dependency, and Directed Technical Change: Evidence from the Auto Industry , 2012, Journal of Political Economy.

[5]  E. Ughetto,et al.  Funding Innovation at Regional Level: An Analysis of a Public Policy Intervention in the Piedmont Region , 2014 .

[6]  A. Grubler,et al.  Global R&D, Market Formation, and Diffusion Investments in Energy Technology Innovation , 2014 .

[7]  R. Boschma,et al.  What drives patent performance of German biotech firms? The impact of R&D subsidies, knowledge networks and their location , 2011 .

[8]  Dirk Czarnitzki,et al.  The Relationship between R&D Collaboration, Subsidies and Patenting Activity: Empirical Evidence from Finland and Germany , 2004 .

[9]  Alessandro Maffioli,et al.  The Impact of National Research Funds: A Regression Discontinuity Approach to the Chilean FONDECYT , 2012 .

[10]  Pierre Azoulay,et al.  Public R&D Investments and Private-Sector Patenting: Evidence from NIH Funding Rules , 2015, The Review of economic studies.

[11]  Elias Einiö R&D Subsidies and Company Performance: Evidence from Geographic Variation in Government Funding Based on the ERDF Population-Density Rule , 2014, Review of Economics and Statistics.

[12]  Ulrich Kaiser,et al.  Long-Run Effects of Public-Private Research Joint Ventures: The Case of the Danish Innovation Consortia Support Scheme , 2012, SSRN Electronic Journal.

[13]  L. Anadón,et al.  Rescue US energy innovation , 2017 .

[14]  P. Aghion,et al.  The Environment and Directed Technical Change , 2009, The American economic review.

[15]  Xulia González,et al.  Do public subsidies stimulate private R&D spending? , 2008 .

[16]  Reinhard Hujer,et al.  Evaluating the Impacts of Subsidies on Innovation Activities in Germany , 2005 .

[17]  J. Foreman-Peck Effectiveness and efficiency of SME innovation policy , 2013 .

[18]  Antoine Dechezleprêtre,et al.  Knowledge Spillovers from Clean and Dirty Technologies , 2014 .

[19]  M. Colombo,et al.  R&D Subsidies and the Performance of High-Tech Start-Ups , 2011 .

[20]  N. Rao,et al.  Do Tax Credits Stimulate R&D Spending? The Effect of the R&D Tax Credit in Its First Decade , 2015 .

[21]  Elena Verdolini,et al.  At Home and Abroad: An Empirical Analysis of Innovation and Diffusion in Energy-Efficient Technologies , 2011 .

[22]  Mauro Mediavilla,et al.  Evaluating the Impact of Public Subsidies on a Firm's Performance: A Quasi-Experimental Approach , 2007 .

[23]  María Consuelo Pazó Martínez,et al.  Do public subsidies stimulate private R&D spending? , 2006 .

[24]  David Popp,et al.  Induced Innovation and Energy Prices , 2001 .

[25]  David S. Lee,et al.  Regression Discontinuity Designs in Economics , 2009 .

[26]  H. Overman What works centre for local economic growth , 2013 .

[27]  Varun Sivaram Unlocking Clean Energy , 2017 .

[28]  David Popp,et al.  THEY DON'T INVENT THEM LIKE THEY USED TO: AN EXAMINATION OF ENERGY PATENT CITATIONS OVER TIME , 2005 .

[29]  Dirk Czarnitzki,et al.  The Relationship between R&D Collaboration, Subsidies and R&D Performance: Empirical Evidence from Finland and Germany , 2007 .

[30]  Instrument Choice in Environmental Policy , 2008, Review of Environmental Economics and Policy.

[31]  Saul Lach,et al.  Do R&D Subsidies Stimulate or Displace Private R&D? Evidence from Israel , 2000 .

[32]  R. Nelson The Simple Economics of Basic Scientific Research , 1959, Journal of Political Economy.

[33]  David Popp,et al.  The effect of new technology on energy consumption , 2001 .

[34]  John Van Reenen,et al.  Do R&D tax credits work? Evidence from a panel of countries 1979–1997 , 2002 .

[35]  Desiderio Romero-Jordán,et al.  Assessment of the public tools used to promote R&D investment in Spanish SMEs , 2014 .

[36]  Adam B. Jaffe,et al.  Building Programme Evaluation into the Design of Public Research‐Support Programmes , 2002 .

[37]  A. Sissoko R&D SUBSIDIES AND FIRM-LEVEL PRODUCTIVITY: EVIDENCE FROM FRANCE , 2011 .

[38]  A. Jaffe,et al.  A tale of two market failures: Technology and environmental policy , 2005 .

[39]  Tom Coupé,et al.  Science Is Golden: Academic R&D and University Patents , 2003 .

[40]  Richard G. Newell,et al.  Environmental and Technology Policies for Climate Mitigation , 2008 .

[41]  Andrea Bonaccorsi,et al.  Do incentives to industrial R&D enhance research productivity and firm growth? Evidence from the Italian case , 2010, Int. J. Technol. Manag..

[42]  Helping the President , 2009, Science.

[43]  Daron Acemoglu,et al.  Transition to Clean Technology , 2016, Journal of Political Economy.

[44]  A. Jaffe,et al.  The impact of R&D subsidy on innovation: evidence from New Zealand firms , 2017 .

[45]  Paolo Piselli,et al.  The Impact of R&D Subsidies on Firm Innovation , 2014 .

[46]  Lee Branstetter,et al.  Japanese Research Consortia: A Microeconometric Analysis of Industrial Policy , 1997 .

[47]  D. Acemoglu,et al.  Transition to Clean Technology , 2014, Journal of Political Economy.

[48]  J. Nishimura,et al.  Subsidy and networking: The effects of direct and indirect support programs of the cluster policy , 2011 .

[49]  D. Popp,et al.  Renewable Energy Policies and Technological Innovation: Evidence Based on Patent Counts , 2008 .

[50]  L. Goulder,et al.  Instrument Choice in Environmental Policy , 2008, Review of Environmental Economics and Policy.

[51]  J. Nishimura,et al.  The effect of participation in government consortia on the R&D productivity of firms: a case study of robot technology in Japan , 2010 .

[52]  K. Ohno Tracking Clean Energy Progress , 2021, Journal of the Atomic Energy Society of Japan.

[53]  J. Van Reenen,et al.  Do Tax Incentives for Research Increase Firm Innovation? An Rd Design for R&D , 2016 .

[54]  Ina Ganguli Saving Soviet Science: The Impact of Grants When Government R&D Funding Disappears , 2017 .

[55]  Cristina Bayona-Sáez,et al.  Assessing the effectiveness of the Eureka Program , 2010 .

[56]  M. Dumont The impact of subsidies and fiscal incentives on corporate R&D expenditures in Belgium (2001-2009) , 2013 .

[57]  Jarle Møen,et al.  Estimating the additionality of R&D subsidies using proposal evaluation data to control for research intentions , 2014 .

[58]  L. Grilli,et al.  Do public subsidies affect the performance of new technology-based firms? The importance of evaluation schemes and agency goals , 2011 .

[59]  Lynn Nielsen-Bohlman,et al.  The toolkit. , 2003, Occasional paper.

[60]  Douglas J. Miller,et al.  Public funds and local biotechnology firm creation , 2014 .

[61]  Susan Athey,et al.  The Econometrics of Randomized Experiments , 2016, 1607.00698.

[62]  James J. Heckman,et al.  Characterizing Selection Bias Using Experimental Data , 1998 .

[63]  S. Afcha Analyzing the Interaction between R&D Subsidies and Firm's Innovation Strategy , 2012 .

[64]  Varun Sivaram,et al.  Venture Capital and Cleantech: The Wrong Model for Energy Innovation , 2016 .

[65]  Beyond surveys: The research frontier moves to the use of administrative data to evaluate R&D grants , 2013 .

[66]  L. Anadón,et al.  Trends in investments in global energy research, development, and demonstration , 2011 .

[67]  Alessandro Maffioli,et al.  Public Support to Firm-Level Innovation: An Evaluation of the FONTEC Program , 2012 .

[68]  J. García‐Quevedo,et al.  Public Subsidies to Business R&D: Do They Stimulate Private Expenditures? , 2005 .

[69]  Petr Hanel,et al.  Evaluating the Impact of R&D Tax Credits on Innovation: A Microeconometric Study on Canadian Firms , 2011 .

[70]  N. Duch,et al.  Evaluating the impact of public subsidies on a firm ’ s performance : a two-stage quasi-experimental approach * , 2009 .

[71]  M. Mazzucato,et al.  Mission-oriented innovation policy and dynamic capabilities in the public sector , 2018, Industrial and Corporate Change.

[72]  Stephen H. Schneider,et al.  Induced technological change and the attractiveness of CO2 abatement policies , 1999 .

[74]  James J. Heckman,et al.  Characterizing Selection Bias Using Experimental Data , 1998 .

[75]  D. Adam Science funders gamble on grant lotteries , 2019, Nature.

[76]  Peter Teirlinck,et al.  Fostering industry-science cooperation through public funding: differences between universities and public research centres , 2012 .

[77]  A. Grubler,et al.  Apples, oranges, and consistent comparisons of the temporal dynamics of energy transitions , 2016 .

[78]  K. Arrow The Economic Implications of Learning by Doing , 1962 .

[79]  Kris Aerts,et al.  Two for the Price of One? On Additionality Effects of R&D Subsidies: A Comparison between Flanders and Germany , 2006 .

[80]  Are Incentives for R&D Effective? Evidence from a Regression Discontinuity Approach , 2014 .

[81]  Davide Fantino,et al.  Evaluating the Efficacy of European Regional Funds for R&D , 2013 .

[82]  Inger Schow Danish Agency for Science Technology and Innovation , 2009 .

[83]  J. Lane Assessing the Impact of Science Funding , 2009, Science.

[84]  Arnulf Grubler,et al.  Energy technology innovation : learning from historical successes and failures , 2013 .

[85]  David Popp,et al.  Economic analysis of scientific publications and implications for energy research and development , 2016, Nature Energy.

[86]  S. Wallsten The Effects of Government-Industry R&D Programs on Private R&D: The Case of the Small Business Innovation Research Program , 2000 .

[87]  T. Brenner,et al.  Joint R&D Subsidies, Related Variety, and Regional Innovation , 2017 .

[88]  Evaluating the efficacy of European regional funds for R&D , 2015 .

[89]  William B. Bonvillian,et al.  Technological Innovation in Legacy Sectors , 2015 .

[90]  Sandro Montresor,et al.  Regional Innovation Policy and Innovative Behaviour: Looking for Additional Effects , 2014 .

[91]  Sabrina T. Howell Financing Innovation: Evidence from R&D Grants , 2016 .