Optimal Estimation when Researcher and Social Preferences are Misaligned

Econometric analysis typically focuses on the statistical properties of fixed estimators and ignores researcher choices. In this article, I approach the analysis of experimental data as a mechanism-design problem that acknowledges that researchers choose between estimators, sometimes based on the data and often according to their own preferences. Specifically, I focus on covariate adjustments, which can increase the precision of a treatment-effect estimate, but open the door to bias when researchers engage in specification searches. First, I establish that unbiasedness is a requirement on the estimation of the average treatment effect that aligns researchers’ preferences with the minimization of the mean-squared error relative to the truth, and that fixing the bias can yield an optimal restriction in a minimax sense. Second, I provide a constructive characterization of all treatment-effect estimators with fixed bias as samplesplitting procedures. Third, I show how these results imply flexible pre-analysis plans for randomized experiments that include beneficial specification searches and offer an opportunity to leverage machine learning. Jann Spiess, Microsoft Research New England, jspiess@stanford.edu. For their guidance I am indebted to Sendhil Mullainathan, Alberto Abadie, Elie Tamer, and Gary Chamberlain. For valuable comments or discussions I also thank Laura Blattner, Avi Feller, Edward Glaeser, Nathaniel Hendren, Simon Jäger, Maximilian Kasy, Lawrence Katz, Scott Kominers, Eben Lazarus, Shengwu Li, Mikkel Plagborg-Møller, Ashesh Rambachan, Jonathan Roth, Elizabeth Santorella, Klaus M. Schmidt, and seminar audiences at Harvard/MIT, MSR, Nortwestern, Princeton, Yale, NYU, Columbia, Chicago, Penn, Penn State, UCL, LSE, Stanford, Queen Mary, CERGE-EI, Copenhagen, UT Austin, and Toronto.