Process Evaluation Insights on Program Implementation

Author(s): Peters, Jane S; McRae, Marjorie R | Abstract: This white paper has the explicit intention to draw lessons learned from the past 30 years of energy efficiency program evaluation in order to facilitate improved program design and implementation going forward. The discussion in this white paper is developed based on interviews with 43 individuals who are either practitioners or users of process and market evaluation. In addition, we obtained references to published materials from our contacts and reviewed conference proceedings dating from 1992 to 2008, which resulted in a review of nearly 100 articles or reports documenting the results of, or commenting on, process and market evaluations.

[1]  Loren Lutzenhiser,et al.  Assessing Markets To Design Programs That More Fully Attack Key Market Barriers and Take Advantage of Market Opportunities - Why Do We Continue To Miss So Many Opportunities? , 2005 .

[2]  Jacqueline Berger,et al.  Transforming Energy Behavior of Households: Evidence from Low-Income Energy Education Programs , 2008 .

[3]  Roland T. Rust,et al.  Service Quality: New Directions in Theory and Practice , 1993 .

[4]  L. Lutzenhiser.  Social and Behavioral Aspects of Energy use , 1993 .

[5]  Edward Vine,et al.  Strategies and policies for improving energy efficiency programs : Closing the loop between evaluation and implementation , 2008 .

[6]  B. Mast.  The Roles of Incentives and Information in DSM Programs , 1996 .

[7]  S. P. Hodges,et al.  How organizational culture influences outcome information utilization. , 1999, Evaluation and program planning.

[8]  Scott Pigg,et al.  How Much Is that Training Program Worth? Quantifying the Value of Training and Other 'Fuzzy' Education Events , 2006 .

[9]  Patricia Gonzales,et al.  How Organizations Implement Evaluation Results , 2007 .

[10]  Mary Kane,et al.  Concept Mapping for Planning and Evaluation , 2006 .

[11]  K. Janda.  Bounded decision making and analytical biases in demand side management , 1994 .

[12]  Brian Coates,et al.  Evaluating Multi-Resource Audit Programs to Demonstrate Sustainability, Payback, and Customer Benefits: Incorporating Non-Energy Benefits (NEBs) , 2001 .

[13]  R. H. Waterman,et al.  In Search of Excellence: Lessons from America's Best-run Companies , 1995 .

[14]  Patricia J. Rogers,et al.  Program theory in evaluation : challenges and opportunities , 2000 .

[15]  E. Vine.  The Integration of Energy Efficiency, Renewable Energy, Demand Response and Climate Change: Challenges and Opportunities for Evaluators and Planners , 2007 .

[16]  Loren Lutzenhiser,et al.  Behavioral Assumptions Underlying California Residential Sector Energy Efficiency Programs , 2009 .

[17]  Carrie Webber.  The Evolution of a Successful Efficiency Program: Energy Savings Bid , 2008 .

[18]  Nina Roberts.  Measuring Program Outcomes: A Practical Approach , 1998 .

[19]  Bobbi Tannenbaum,et al.  Training: Designing for Lasting Change , 2006 .

[20]  Joseph H. Eto,et al.  A scoping study on energy-efficiency market transformation by California Utility DSM Programs , 1996 .

[21]  Marc R. Ledbetter,et al.  Seven Years Since SERP: Successes and Setbacks in Technology Procurement , 2002 .

[22]  C. Weiss,et al.  TRUTH TESTS AND UTILITY TESTS: DECISION-MAKERS' FRAMES OF REFERENCE FOR SOCIAL SCIENCE RESEARCH* , 1980 .

[23]  Lori Megdal,et al.  Learning from Successive Evaluations of a Low-Income Program , 2005 .

[24]  R. N. Elliott,et al.  American Council for an Energy-Efficient Economy , 2002 .

[25]  William B. Kallock.  The Importance of Field Observations in Evaluations On-site investigations are still the best way to know what is happening in the field. , 2005 .

[26]  A. Matthews.  Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation , 2006 .

[27]  Nikhil Gandhi,et al.  Product Selection—A Forgotten Vital Component of Program Design , 2007 .

[28]  A. T. Cate,et al.  Technology procurement as a market transformation tool , 1998 .

[29]  Jane S. Peters,et al.  China Motor Systems Energy Conservation Project Process Evaluation A Cross-Cultural Process Evaluation of a Cross-Cultural Program , 2005 .

[30]  P. Grasso.  What Makes an Evaluation Useful? Reflections from Experience in Large Organizations , 2003 .

[31]  Evaluating Consumer Energy Aggregations : A Policy Perspective , 2000 .

[32]  P. J. Spinney,et al.  DSM process evaluation: A guidebook to current practice , 1992 .

[33]  Lisa Wyatt Knowlon,et al.  The Logic Model Guidebook: Better Strategies for Great Results , 2008 .

[34]  Peter Dahler-Larsen.  Beyond non-utilization of evaluations: An institutional perspective , 1998 .

[35]  M. Sami Khawaja,et al.  Using Simplified Pattern Matching to Define Program Theory and Assess Program Effectiveness—Application to a Green Buildings Program , 2007 .

[36]  Ville Valovirta.  Evaluation Utilization as Argumentation , 2002 .

[37]  What developers think about district energy: A mental models approach , 2006 .

[38]  Melvin M. Mark,et al.  The Mechanisms and Outcomes of Evaluation Influence , 2004 .