Panel experiments and dynamic causal effects: A finite population perspective

In panel experiments, we randomly assign units to different interventions, measuring their outcomes, and repeating the procedure in several periods. Using the potential outcomes framework, we define finite population dynamic causal effects that capture the relative effectiveness of alternative treatment paths. For a rich class of dynamic causal effects, we provide a nonparametric estimator that is unbiased over the randomization distribution and derive its finite population limiting distribution as either the sample size or the duration of the experiment increases. We develop two methods for inference: a conservative test for weak null hypotheses and an exact randomization test for sharp null hypotheses. We further analyze the finite population probability limit of linear fixed effects estimators. These commonly‐used estimators do not recover a causally interpretable estimand if there are dynamic causal effects and serial correlation in the assignments, highlighting the value of our proposed estimator.

[1]  Andrew Goodman-Bacon Difference-in-Differences with Variation in Treatment Timing , 2018, Journal of Econometrics.

[2]  David A. Freedman,et al.  On regression adjustments to experimental data , 2008, Adv. Appl. Math..

[3]  J. Robins Correcting for non-compliance in randomized trials using structural nested mean models , 1994 .

[4]  D. V. Lindley,et al.  Randomization Analysis of Experimental Data: The Fisher Randomization Test Comment , 1980 .

[5]  G. Imbens,et al.  Double-Robust Identification for Causal Panel Data Models , 2019, SSRN Electronic Journal.

[6]  S. Murphy,et al.  Optimal dynamic treatment regimes , 2003 .

[7]  G. Imbens,et al.  Matrix Completion Methods for Causal Panel Data Models , 2017, Journal of the American Statistical Association.

[8]  Mélanie Frappier,et al.  The Book of Why: The New Science of Cause and Effect , 2018, Science.

[9]  Iavor I Bojinov,et al.  Avoid the Pitfalls of A/B Testing , 2020 .

[10]  Joseph Suresh Paul,et al.  Matrix Completion Methods , 2019, Regularized Image Reconstruction in Parallel MRI with MATLAB®.

[11]  D. Horvitz,et al.  A Generalization of Sampling Without Replacement from a Finite Universe , 1952 .

[12]  J. Heckman,et al.  Dynamic Treatment Effects. , 2016, Journal of econometrics.

[13]  Fredrik Sävje,et al.  AVERAGE TREATMENT EFFECTS IN THE PRESENCE OF UNKNOWN INTERFERENCE. , 2017, Annals of statistics.

[14]  M. Hashem Pesaran,et al.  Impulse response analysis in nonlinear multivariate models , 1996 .

[15]  Andrew Goodman-Bacon Difference-in-differences with variation in treatment timing , 2021, Journal of Econometrics.

[16]  D. Rubin,et al.  Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction , 2016 .

[17]  Gary Charness,et al.  Journal of Economic Behavior & Organization , 2022 .

[18]  T. Speed,et al.  On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9 , 1990 .

[19]  P. Ding A paradox from randomization-based causal inference , 2014, 1402.0142.

[20]  M. Reich,et al.  Corrigendum: Credible Research Designs for Minimum Wage Studies: A Response to Neumark, Salas, and Wascher , 2018 .

[21]  Hairul Azlan Annuar,et al.  Foreign investors' interests and corporate tax avoidance: Evidence from an emerging economy , 2015 .

[22]  Larry Samuelson,et al.  Building rational cooperation , 2006, J. Econ. Theory.

[23]  J. Wooldridge Fixed-Effects and Related Estimators for Correlated Random-Coefficient and Treatment-Effect Panel Data Models , 2005, Review of Economics and Statistics.

[24]  H. Robbins,et al.  Asymptotically efficient adaptive allocation rules , 1985 .

[25]  James J. Heckman,et al.  Estimating the Technology of Cognitive and Noncognitive Skill Formation , 2010, Econometrica : journal of the Econometric Society.

[26]  Xavier d'Haultfoeuille,et al.  Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects , 2018, American Economic Review.

[27]  J. Droege ESTIMATING THE RETURNS TO SCHOOLING , 2016 .

[28]  M. Lechner The Estimation of Causal Effects by Difference-in-Difference Methods , 2011 .

[29]  S. Murphy,et al.  Assessing Time-Varying Causal Effect Moderation in Mobile Health , 2016, Journal of the American Statistical Association.

[30]  Kosuke Imai,et al.  When Should We Use Unit Fixed Effects Regression Models for Causal Inference with Longitudinal Data? , 2019, American Journal of Political Science.

[31]  M. Browning,et al.  Modelling income processes with lots of heterogeneity , 2010 .

[32]  Manuel Arellano,et al.  Nonlinear Panel Data Analysis , 2011 .

[33]  H. Robbins Some aspects of the sequential design of experiments , 1952 .

[34]  P. Hall,et al.  Martingale Limit Theory and its Application. , 1984 .

[35]  Charles Bellemare,et al.  Simulating power of economic experiments: the powerBBK package , 2016 .

[36]  Xavier Jaravel,et al.  Revisiting Event Study Designs , 2017 .

[37]  Susan Athey,et al.  Design-based analysis in Difference-In-Differences settings with staggered adoption , 2021 .

[38]  J. List,et al.  The Dozen Things Experimental Economists Should Do (More of) , 2019, Southern Economic Journal.

[39]  J. Robins,et al.  Estimation of the Causal Effect of a Time-Varying Exposure on the Marginal Mean of a Repeated Binary Outcome , 1999 .

[40]  Wei Chu,et al.  A contextual-bandit approach to personalized news article recommendation , 2010, WWW '10.

[41]  N. Schork,et al.  The n-of-1 clinical trial: the ultimate strategy for individualizing medicine? , 2011, Personalized medicine.

[42]  Michael E. Sobel,et al.  What Do Randomized Studies of Housing Mobility Demonstrate? , 2006 .

[43]  P. Holland Statistics and Causal Inference , 1985 .

[44]  Susan Athey,et al.  Sampling‐Based versus Design‐Based Uncertainty in Regression Analysis , 2017, Econometrica.

[45]  Y. Ben-Porath The Production of Human Capital and the Life Cycle of Earnings , 1967, Journal of Political Economy.

[46]  Peter Hull Estimating Treatment Effects in Mover Designs , 2018, 1804.06721.

[47]  J Mark,et al.  The Construction and Analysis of Adaptive Group Sequential Designs , 2008 .

[48]  D. Rubin Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[49]  R. Blundell,et al.  Earnings and Consumption Dynamics: A Nonlinear Panel Data Framework , 2015, SSRN Electronic Journal.

[50]  Liyang Sun,et al.  Estimating Dynamic Treatment Effects in Event Studies With Heterogeneous Treatment Effects , 2018, Journal of Econometrics.

[51]  M. Nerlove,et al.  Biases in dynamic models with fixed effects , 1988 .

[52]  Stéphane Bonhomme,et al.  Nonlinear Panel Data Estimation via Quantile Regressions , 2015 .

[53]  Stan Hurn Panel Data Econometrics , 2010 .

[54]  I NICOLETTI,et al.  The Planning of Experiments , 1936, Rivista di clinica pediatrica.

[55]  D. Rubin [On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.] Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies , 1990 .

[56]  Stanley H. Cohen,et al.  Design and Analysis , 2010 .

[57]  Z. Griliches Estimating the Returns to Schooling: Some Econometric Problems. , 1977 .

[58]  James J. Heckman,et al.  Interpreting the Evidence on Life Cycle Skill Formation , 2005, SSRN Electronic Journal.

[59]  J. Robins A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect , 1986 .

[60]  Kosuke Imai,et al.  On the Use of Two-Way Fixed Effects Regression Models for Causal Inference with Panel Data , 2020, Political Analysis.

[61]  Alex J. Chin,et al.  Central limit theorems via Stein's method for randomized experiments under interference , 2018, 1804.03105.

[62]  Oscar Kempthorne,et al.  THE RANDOMIZATION THEORY OF' EXPERIMENTAL INFERENCE* , 1955 .

[63]  Marc Nerlove,et al.  Further evidence on the estimation of dynamic economic relations from a time series of cross-sections , 1971 .

[64]  Cheng Hsiao,et al.  Formulation and estimation of dynamic models using panel data , 1982 .

[65]  Charles Bellemare,et al.  Statistical Power of within and Between-Subjects Designs in Economic Experiments , 2014, SSRN Electronic Journal.

[66]  J. List,et al.  The Dozen Things Experimental Economists Should Do (More Of) , 2019, SSRN Electronic Journal.

[67]  Peter M. Aronow,et al.  Estimating Average Causal Effects Under Interference Between Units , 2013, 1305.6156.

[68]  Kelly W. Zhang,et al.  Inference for Batched Bandits , 2020, NeurIPS.

[69]  Stefan Wager,et al.  Confidence intervals for policy evaluation in adaptive experiments , 2021, Proceedings of the National Academy of Sciences.

[70]  J M Robins,et al.  Marginal Mean Models for Dynamic Regimes , 2001, Journal of the American Statistical Association.

[71]  Shuai Li,et al.  Collaborative Filtering Bandits , 2015, SIGIR.

[72]  P. Ding,et al.  General Forms of Finite Population Central Limit Theorems with Applications to Causal Inference , 2016, 1610.04821.

[73]  Neil Shephard,et al.  Time Series Experiments and Causal Estimands: Exact Randomization Tests and Trading , 2017, Journal of the American Statistical Association.

[74]  Susan Athey,et al.  Design-Based Analysis in Difference-in-Differences Settings with Staggered Adoption , 2018, Journal of Econometrics.

[75]  Susan Athey,et al.  When Should You Adjust Standard Errors for Clustering? , 2017, The Quarterly Journal of Economics.

[76]  Judea Pearl,et al.  Causal Inference , 2010 .

[77]  Sukjin Han Identification in Nonparametric Models for Dynamic Treatment Effects , 2018, Journal of Econometrics.

[78]  P. Hall,et al.  Martingale Limit Theory and Its Application , 1980 .