Using Simplified Pattern Matching to Define Program Theory and Assess Program Effectiveness—Application to a Green Buildings Program

Although efforts have increased to apply theory-based methods to evaluate energy-efficiency programs, implementation has been limited. One approach used in other fields is combining concept mapping and pattern matching to elicit and test program theories. This provides a means of comparing the program designers’ program theory perceptions with those of targeted market actors. Concept mapping typically employs techniques to cluster either metric or non-metric data based on similarities, thus providing a map that groups concepts perceived to be similar. A program implementers’ concept map can be used to reveal their perceptions about assumed linkages between program activities and expected outcomes (essentially the program theory). To test the theory, pattern matching is used to compare designer/implementer ratings of a program’s expected effects on key outcomes with findings based on data from participants and differences observed between participants and non-participants. This paper describes how we developed and applied a simplified version of this approach to articulate a program theory and provide insights on the theory’s validity for a specific energy-efficient green building program. Program designer/implementer information was gathered from a one-day workshop, with follow-up submittals. The second step occurred during program evaluation when program participants were surveyed and asked to rate the barriers and the effectiveness of specific program services in reducing the barriers. These steps helped elucidate the program theory at project outset and provided the mechanism to compare anticipated program effects with actual outcomes and, therefore, test the consistency of the program theory against the reality of program outcomes.