External Validity and Partner Selection Bias

Program evaluation often involves generalizing internally-valid site-speci…c estimates to a dierent population or environment. While there is substantial evidence on the internal valid- ity of non-experimental relative to experimental estimates (e.g. Lalonde 1986), there is little quantitative evidence on the external validity of site-speci…c estimates, because identical treat- ments are rarely evaluated in multiple settings. This paper examines a remarkable series of 14 energy conservation …eld experiments run by a company called OPOWER, involving 550,000 households in dierent cities across the U.S. Despite the availability of potentially-promising individual-level controls, we show that the unexplained variation in treatment eects across sites is both statistically and economically signi…cant. Furthermore, we show that the electric utilities that partner with OPOWER dier systematically on characteristics that are correlated with the treatment eect, providing evidence of a "partner selection bias" that is analogous to biases caused by individual-level selection into treatment. We augment this result in a dierent context by showing that partner micro…nancial institutions (MFIs) that carry out randomized experiments appear to be selected on observable characteristics from the global pool of MFIs. Finally, we propose a statistical test for parameter heterogeneity at "sub-sites" within a site that provides suggestive evidence on whether site-speci…c estimates can be generalized.