Problems and Methods in the Study of Politics: The illusion of learning from observational research

Theory-testing occupies a central place within social science, but what kinds of evidence count toward a meaningful test of a causal proposition? We offer two analytic results that have important implications for the relative merits of observational and experimental inquiry. The first result addresses the optimal weight that should be assigned to unbiased experimental research and potentially biased observational research. We find that unless researchers have prior information about the biases associated with observational research, observational findings are accorded zero weight regardless of sample size, and researchers learn about causality exclusively through experimental results. The second result describes the optimal allocation of future resources to observational and experimental research, given different marginal costs for each type of data collection and a budget constraint. Under certain conditions (e.g., severe budget constraints and prohibitive costs of experimental data), it is optimal to allocate one’s entire budget to observational data. However, so long as one harbors some uncertainty about the biases of observational research, even an infinite supply of observational data cannot provide exact estimates of causal parameters. In the absence of theoretical or methodological breakthroughs, the only possibility for further learning comes from experiments, particularly experiments with strong external validity.