The emperor has no clothes: using interrupted time series designs to evaluate social policy impact

The most popular quasi-experimental strategy for evaluating the aggregate impact of changes in law and other social policies is the univariate interrupted time series design (ITSD). In practice, the internal validity of this approach has been greatly exaggerated and its users have largely ignored or minimized its flaws, including: (1) its general inability to rule out alternative explanations, (2) the use of a single or small number of arbitrarily chosen 'control' or comparison jurisdictions, (3) arbitrary definition of the endpoints of the time series evaluated, (4) an inability to specify exactly when the intervention’s impact is supposed to be felt, raising problems of the falsifiability of the efficacy hypothesis, and (5) an atheoretical specification of the ARIMA impact model. Data pertaining to the 1976 Washington, D.C., handgun ban are analyzed to illustrate these problems. Authors of a previous evaluation concluded that the ban reduced homicides; this conclusion collapses when any one of several valid changes in analytic strategy are made. Further, when 'bogus intervention' points are specified, corresponding to nonexistent policy interventions, results as strong as those obtained by the original authors are obtained. Finally, when the same ITSD strategy is applied to an example of gun 'decontrol,' a gun law repeal exactly opposite in character to that of the D.C. law, the same appearance of a homicide-reducing impact is generated. It is concluded that the univariate ITSD approach is of little value for policy assessment, because it can so easily be manipulated to generate results compatible with a researcher's preconceived biases. This is a revised version of a paper presented at the annual meeting of the American Society of Criminology in Phoenix, Arizona, October 30, 1993. A portion of this paper was published in 1996 in Law & Society Review.