Application of theory-based evaluation for the critical analysis of national biofuel policy: A case study in Malaysia.

Theory-based evaluation (TBE) is an effectiveness assessment technique that critically analyses the theory underlying an intervention. Whilst its use has been widely reported in the area of social programmes, it is less applied in the field of energy and climate change policy evaluations. This paper reports a recent study that has evaluated the effectiveness of the national biofuel policy (NBP) for the transport sector in Malaysia by adapting a TBE approach. Three evaluation criteria were derived from the official goals of the NBP, those are (i) improve sustainability and environmental friendliness, (ii) reduce fossil fuel dependency, and (iii) enhance stakeholders' welfare. The policy theory underlying the NBP has been reconstructed through critical examination of the policy and regulatory documents followed by a rigorous appraisal of the causal link within the policy theory through the application of scientific knowledge. This study has identified several weaknesses in the policy framework that may engender the policy to be ineffective. Experiences with the use of a TBE approach for policy evaluations are also shared in this report.

[1]  J. Houghton,et al.  Climate Change 2013 - The Physical Science Basis: Working Group I Contribution to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change , 2014 .

[2]  Mirjam Harmelink,et al.  The theory-based policy evaluation method applied to the ex-post evaluation of climate change policies in the built environment in the Netherlands , 2005 .

[3]  Lars J Nilsson,et al.  Theory-based policy evaluation of 20 energy efficiency instruments , 2008 .

[4]  S. Solomon.  The Physical Science Basis : Contribution of Working Group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change , 2007 .

[5]  K. Owen,et al.  Automotive Fuels Reference Book , 1995 .

[6]  Stewart I. Donaldson,et al.  Program Theory-Driven Evaluation Science: Strategies and Applications , 2007 .

[7]  Vincent Mahieu,et al.  Well-to-wheels analysis of future automotive fuels and powertrains in the european context , 2004 .

[8]  Stewart I. Donaldson,et al.  Theory-driven Evaluation in Action: Lessons from a $20 Million Statewide Work and Health Initiative , 2003 .

[9]  Masson-Delmotte,et al.  The Physical Science Basis , 2007 .

[10]  M. Chin.  Biofuels in Malaysia: An analysis of the legal and institutional framework , 2011 .

[11]  Jacinto F. Fabiosa,et al.  Use of U.S. Croplands for Biofuels Increases Greenhouse Gases Through Emissions from Land-Use Change , 2008, Science.

[12]  Stewart I. Donaldson,et al.  Using program theory–driven evaluation science to crack the Da Vinci code , 2005 .

[13]  F. Leeuw,et al.  Reconstructing Program Theories: Methods Available and Problems to be Solved , 2003 .

[14]  Keat Teong Lee,et al.  Malaysian palm oil: Surviving the food versus fuel dispute for a sustainable future , 2009 .

[15]  A. Schäfer.  REGULARITIES IN TRAVEL DEMAND: AN INTERNATIONAL PERSPECTIVE , 2000 .

[16]  R. Pawson,et al.  Theory-driven evaluation of public health programmes , 2009 .

[17]  P. H. Bly,et al.  Changing patterns of urban travel Part 2. public transport and future patterns of travel , 1986 .

[18]  Tick Hui Oh,et al.  Green progress and prospect in Malaysia , 2011 .

[19]  Keat Teong Lee,et al.  Will biofuel projects in Southeast Asia become white elephants , 2010 .

[20]  A. Mohamed,et al.  A comparative study on the energy policies in Japan and Malaysia in fulfilling their nations’ obligations towards the Kyoto Protocol , 2009 .

[21]  Michael Q. Wang,et al.  The Energy Balance of Corn Ethanol: An Update , 2002 .

[22]  Mareike Lange,et al.  The GHG Balance of Biofuels Taking into Account Land Use Change (Power Point) , 2011 .

[23]  Keith A. Smith,et al.  N 2 O release from agro-biofuel production negates global warming reduction by replacing fossil fuels , 2007 .

[24]  H. White.  Theory-Based Impact Evaluation , 2009 .

[25]  Francesco Cherubini,et al.  Energy- and greenhouse gas-based LCA of biofuel and bioenergy systems: Key issues, ranges and recommendations , 2009 .

[26]  C. Weiss.  How Can Theory-Based Evaluation Make Greater Headway? , 1997 .

[27]  W. Clark,et al.  Biofuels and Sustainable Development , 2008 .

[28]  M. Kelly,et al.  Evidence-based Public Health: Effectiveness and efficiency , 2010 .

[29]  A. V. Amstel,et al.  The land cover and carbon cycle consequences of large-scale utilizations of biomass as an energy source , 1996 .

[30]  Subhash Bhatia,et al.  Current status and policies on biodiesel industry in Malaysia as the world's leading producer of palm oil , 2009 .

[31]  Carol H. Weiss,et al.  Nothing as Practical as Good Theory : Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families , 2011 .

[32]  G. Timilsina,et al.  Transport sector CO2 emissions growth in Asia: Underlying factors and policy options , 2009 .

[33]  A. Faaij,et al.  Different palm oil production systems for energy purposes and their greenhouse gas implications , 2008 .

[34]  E. Suchman.  Evaluative Research: Principles and Practice in Public Service and Social Action Progr , 1968 .

[35]  Chris L. S. Coryn,et al.  A Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009 , 2011 .

[36]  Teoh Cheng Hai,et al.  THE PALM OIL INDUSTRY IN MALAYSIA From Seed to Frying Pan , 2002 .

[37]  Astrid Brousselle,et al.  Program theory evaluation: Logic analysis. , 2011, Evaluation and program planning.

[38]  Haji Hassan Masjuki,et al.  A review on energy pattern and policy for transportation sector in Malaysia , 2012 .

[39]  M. Patton,et al.  Evaluation: A Systematic Approach , 1980 .

[40]  Amir F.N Abdul-Manan,et al.  Ex-Post Critical Evaluations of Energy Policies in Malaysia from 1970 to 2010: A Historical Institutionalism Perspective , 2015 .

[41]  P. H. Bly,et al.  Changing patterns of urban travel , 1986 .

[42]  Huey-Tsyh Chen,et al.  Practical program evaluation : assessing and improving planning, implementation, and effectiveness , 2005 .

[43]  K. Owen,et al.  AUTOMOTIVE FUELS REFERENCE BOOK. SECOND EDITION , 1995 .

[44]  S. Polasky,et al.  Land Clearing and the Biofuel Carbon Debt , 2008, Science.

[45]  G. Lopez,et al.  Biofuels - At what cost? Government support for biodiesel in Malaysia , 2008 .

[46]  Lee Wei Chang,et al.  A detailed survey of the palm and biodiesel industry landscape in Malaysia , 2014 .

[47]  T. Searchinger,et al.  Land-Use Change Greenhouse Gases Through Emissions from Use of U.S. Croplands for Biofuels Increases , 2008 .

[48]  Mark W. Lipsey,et al.  Evaluation: A Systematic Approach , 1979 .

[49]  D. Pimentel,et al.  Ethanol Production Using Corn, Switchgrass, and Wood; Biodiesel Production Using Soybean and Sunflower , 2005 .

[50]  Michael Quinn Patton,et al.  Utilization-Focused Evaluation , 1979 .