Detecting Spillover Effects: Design and Analysis of Multilevel Experiments

Interpersonal communication presents a methodological challenge and a research opportunity for researchers involved in field experiments. The challenge is that communication among subjects blurs the line between treatment and control conditions. When treatment effects are transmitted from subject to subject, the stable unit treatment value assumption (SUTVA) is violated, and comparison of treatment and control outcomes may provide a biased assessment of the treatment’s causal influence. Social scientists are increasingly interested in the substantive phenomena that lead to SUTVA violations, such as communication in advance of an election. Experimental designs that gauge SUTVA violations provide useful insights into the extent and influence of interpersonal communication. This article illustrates the value of one such design, a multilevel experiment in which treatments are randomly assigned to individuals and varying proportions of their neighbors. After describing the theoretical and statistical underpinnings of this design, we apply it to a large-scale voter-mobilization experiment conducted in Chicago during a special election in 2009 using social-pressure mailings that highlight individual electoral participation. We find some evidence of within-household spillovers but no evidence of spillovers across households. We conclude by discussing how multilevel designs might be employed in other substantive domains, such as the study of deterrence and policy diffusion.